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I  INTRODUCTION 


INTRODUCTION 


PURPOSE 


This  study  was  conducted  at  the  behest  of  the  U.S.  Army  Armament  Research 
and  Development  Command  (ARRADCOM),  Dover,  New  Jersey,  for  the 
purposes  of: 

Projecting  future  trends  in  electronic  process  control  (EPC)  systems 
development  and  relating  those  trends  to  the  reliability  and  mainte- 
nance of  such  systems  during  long-term  storage. 

Determining  what  private  industrial  experiences  relating  to  the  long- 
term  storage  of  electronic  systems  could  be  applied  to  the  Army 
ammunition  plant  problem. 

Outlining  a  framework  for  the  maintenance  of  electronic  process 
control  systems  in  long-term  storage. 

Determining  the  status  of  layaway  plans  at  selected  Army  ammunition 
plants. 

The  U.S.  Army  Armament  Materiel  Command  (ARRCOM)  manages  the  Army's 
munitions  production  base.  ARRCOM's  headquarters  are  in  Rock  Island, 
Illinois. 
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SCOPE 


The  information  gathered  and  analyzed  is  directed  toward  procuring  new 
electronic  process  control  systems,  upgrading  existing  systems,  and  providing  a 
guide  for  laying  away  all  systems. 

Trends  in  maintenance  methodologies  are  presented  with  an  assessment  of 
their  impact  on  the  dormant  state  problem. 

Maintenance  personnel  sources  and  training  are  discussed  as  they  relate  to  the 
operating  contractors'  role. 

Eight  appendices  are  included  in  support  of  this  study: 

Appendix  A  lists  the  outside  sources  for  this  report  and  also  includes  a 
bibliography. 

Appendix  B  contains  the  definition  of  terms  in  the  process  control  field. 

Appendix  C  lists  professional  services  companies  in  the  computer  field 
specializing  in  manufacturing.  These  companies  provide  consulting, 
software  packages,  training,  facilities  management,  and  programming 
services. 

Appendix  D  lists  the  larger  computer  maintenance  service  companies. 

Appendix  E  includes  tests  of  a  new  product  discovered  during  the 
research  for  this  study,  ^hich  is  effective  in  inhibiting  corrosion  in 
factory  areas  with  high  humidity  and  corrosion  atmospheres. 

Appendix  F  is  a  set  of  GAO  guidelines  for  entering  into  software 
contracts.  These  guidelines,  which  refer  to  government  only,  should  be 
useful  for  any  systems  contract.     In  general,  software  produced  for 


federal  agencies  has  not  been  very  useful  as  it  was  originally  delivered 
to  the  government,  as  shown  in  Exhibit  1-1.  These  guidelines  are 
suggested  to  increase  the  return  on  the  software  investnnent.  (Note: 
There  are  no  comparable  statistics  for  the  private  sector  nor  any  GAO 
follow-up  study  on  the  procurement  results  after  this  initial  study.) 

Appendix  G  lists  the  type  of  companies  interviewed  for  this  study  and 
the  number  of  telephone  and  on-site  interviews  completed  in  the  study. 

Appendix  H  contains  the  interview  guides  used  for  the  study. 
C.  METHODOLOGY 

•  Data  gathering  was  accomplished  by  29  interviews  with  vendors  and  users  of 
electronic  process  control  systems. 

Users  were  engineers  who  were  knowledgeable  in  EPC  systems  and 
vendors  of  electronic  process  control  systems. 

Forty  percent  of  the  interviews  were  done  on-site  and  60%  were  by 
telephone. 

•  Seven  Army  ammunition  plants  were  visited  for  at  least  one  day,  and  were 
reported  on. 

A  visit  was  made  to  the  Joliet  Army  Ammunition  Plant  in  conjunction 
with  a  reactivation  cycle. 

•  An  extensive  literature  search  was  conducted  over  several  commercial  data 
bases  for  literature  pertaining  to  dormancy  of  high-technology  equipment,  but 
little  available  data  was  directly  applicable  to  the  immediate  problem. 
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EXHIBIT  1-1 


NINE  FEDERAL  AGENCY  SOFTWARE  DEVELOPMENT 
CONTRACTS  TOTALING  $6.8  MILLION: 
WHERE  THE  MONEY  WENT 


SOFTWARE  THAT  COULD  BE 
USED  AFTER  CHANGES 
($198,000  out  of  $6.8  MILLION) 

SOFTWARE  THAT  COULD 
BE  USED  AS  DELIVERED 
($119,000  out  of  $6.8  MILLION) 


SOURCE:  GAG  REPORT:  FGMSD-80-4 
DATED:  NOVEMBER  9,  1  979 


OOUO 


Included  in  the  search  were  DIALOGUE  and  NTIS. 

Pertinent  material  was  located  in  previous  INPUT  studies  which  will  be 
referenced  throughout  the  study. 


-5- 


INPUT 


-6- 


II    EXECUTIVE  SUMMARY 


li        EXECUTIVE  SUMMARY 


•  This  report  outlines  the  trends  in  electronic  process  control  (EPC)  systems  as 
they  impact  U.S.  Army  ammunition  plants. 

The  technical  advances  !n  microcircuitry  and  microprocessors,  for 
example,  have  a  significant  impact  on  planning  for  periods  when  the 
plan+s  are  dormant. 

This  impact  will  result  in  replacing  centralized  control  with  decentral- 
ized control  at  the  plant  and  workstation. 

•  The  report  further  examines  the  trends  in  maintenance  service  which  //ill 
profoundly  affect  the  Army's  planning  for  dormant  installations. 

These  trends  will  have  an  impact  which  vvill  be  felt  by  the  Army  in 
planning  for  and  supplying  spare  parts,  obtaining  technical  support,  and 
training. 

•  With  these  trends  as  a  background,  the  following  report  by  INPUT  will  detail 
the  present  status  of  U.S.  Army  ammunition  plants. 

Differences  between  these  plants  and  the  typical  manufacturing  plant 
will  be  explored. 
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•  The  planning  process  for  EPC  systenns  will  be  discussed  with  specific  recom- 
mendations. 

Finally,  commercial  dormancy  experience  will  be  discussed  to  deter- 
mine if  the  Army  can  benefit  from  the  experience  in  the  commercial 
sector. 

A.  CONCLUSIONS 

I.       TRENDS  IN  ELECTRONIC  PROCESS  CONTROL  SYSTEMS 

•  Electronic  process  control  systems  have  completed  an  evolutionary  cycle. 

Initially,  process  control  was  located  at  the  site  of  the  equipment  under 
the  direction  of  individuals  and  instruments. 

Large-scale  computers,  in  the  1960s,  brought  change  as  control  became 
centralized  at  some  remote  control  station. 

The  advent  of  microprocessors  and  inexpensive  computers,  in  the  1970s, 
has  resulted  in  decision-making  and  control  being,  once  again,  decen- 
tralized to  the  field. 

•  The   key  factor,   inexpensive  intelligence  and  control  devices,   is  greatly 
changing  the  technology,  planning,  and  approach  to  EPC  systems. 

The  distribution  of  intelligence  end  control  devices  to  the  workstation 
will  decrease  the  need  for  redundant  hardware  because  individual 
failures  do  not  disrupt  the  total  process. 

The  main  CPU  will  serve  less  as  the  key  component  of  the  control 
system  as  controls  will  be  located  at  the  machine  site. 
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A  single  failure  will  not  have  the  magnitude  of  effect  in  this  new 
environment. 

Systems  will  be  easier  to  maintain  because  the  individual  system's 
elements  will  be  increasingly  more  standardized  and  more  easily 
replaceable. 

TRENDS  IN  COMPUTER  INDUSTRY  MAINTENANCE  SERVICE 

All  costs  of  providing  maintenance  service  are  increasing,  while  the  supply  of 
skilled  personnel  is  decreasing. 

EPC  vendors  are  seeking  to  provide  quality  service  by  concentrating 
their  personnel  in  a  central  or  regional  location  and  having  the 
customer  come  to  this  service  center  in  one  fashion  or  another. 

This  information  came  forth  clearly  from  the  interviews  con- 
ducted in  this  study. 

Providing  service  from  this  service  center  via  communication  lines  and 
having  the  user  participate  in  the  service  process  is  a  viable  means  of 
increasing  the  field  engineers'  productivity  and  of  satisfying  the 
customer's  requirements. 

Using  the  service  center,  called  a  remote  diagnostic  center,  as  the 
testing  center,  EPC  systems  at  ammunition  plants  can  be  reactivated 
and  tested,  and,  as  a  result,  skilled  computer  personnel  will  operate  the 
system  from  this  remote  diagnostic  center. 

The  increasing  use  of  self-diagnostic  hardware  and  software  will  help  service 
personnel  to  detect  defective  components  before  a  bad  component  can  shut 
down  a  system. 
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3. 


U.S.  ARMY  AMMUNITION  PLANTS'  EPCS  STATUS 


•  The  operating  contractors  of  the  ammunition  plants  visited  by  INPUT  were 
aware  of  many  of  the  implications  of  keeping  electronic  systems  dormant  for 
long  periods,  but  few  have  laid  out  comprehensive  plans  for  addressing  the 
effects  of  dormancy. 

The  impetus  provided  by  ARRADCOM's  direction  and  funding  has 
served  to  pinpoint  the  major  problems  and  is  continuing  to  assist  in 
finding  solutions,  but  these  problems  are  not  of  the  highest  priority  for 
the  operating  contractors. 

It  should  be  realized,  however,  that  the  contractors  are  respon- 
sible for  the  entire  plant,  and  the  automated  lines  are  only  a 
portion  of  the  plant's  total  production  capability. 

On  occasion  the  contractors  gave  the  impression  that  the  automated 
line  just  complicated  the  maintenance  and  mobilization  problems. 

•  U.S.  Army  ammunition  plants  are  different  from  commercial  manufacturing 
complexes.  Army  ammunition  plants  do  not  run  at  full  production  all  of  the 
time  as  do  commercial  plants.  Ammunition  plants  operate  at  full  production, 
at  partial  production,  or  are  dormant  during  their  existence. 

Currently,  INPUT  concludes,  these  differences  are  not  always  con- 
sidered in  the  design  of  the  plant  which  is  a  problem  being  addressed  by 
ARRADCOM  in  its  study,  MMT  4322. 

It  appears  to  INPUT  that  the  automated  lines,  even  though  different, 
are  designed  in  the  same  manner  as  industrial  production  lines  -  to 
operate  24  hours  per  day,  365  days  per  year;  the  ARRADCOM  study 
should  shed  more  light  on  this  potential  problem. 
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4. 


LONG-TERM  (20  YEAR)  PLANNING 


•  Planning  for  a  five-year  period  seems  to  be  too  short  because  it  takes  longer 
to  design,  build,  install,  and  test  a  complex  EPC  system. 

EPC  systems  for  Army  ammunition  plants  are  designed  for  20  years 
and,  therefore,  planning  should  include  the  lifetime  of  the  system  in 
order  to  maximize  the  return  on  the  investment. 

•  °lanning  should  include  ail  the  plants  as  well  as  the  longevity  (20  years)  of  the 
plants. 

There  has  not  been  a  consistent  attempt  to  obtain  leverage  with  any 
particular  manufacturer  and  vendors  vary  from  plant  to  plant  just  as 
systems  vary.  System  replication  was  found,  however,  in  at  least  two 
cases,  the  Joliet  Army  Ammunition  Plant  and  the  Volunteer  Army 
Ammunition  Plant. 

•  The  multiplicity  of  contractors  involved  in  each  EPC  system  spreads  the 
responsibility  so  that  conflicts  and  crossed  communications  can  occur  and  can 
lead  to  a  lengthy  design-to-operation  cycle. 

3y  the  time  some  systems  become  operational,  the  hardware  is 
obsolete. 

If  the  line  requires  six  years  from  design  to  implementation  the 
com.puter  could  be  phased  out  by  the  manufacturer. 

5.        CO.MA.AERCIAL  DORMANCY  EXPERIENCE 

•  Some  of  the  commercial  users  of  EPC  systems,  who  were  interviewed, 
experienced  varying  periods  of  dormancy  but  none  experienced  the  1 5  to  20 
years  facing  the  operating  contractors  at  the  ammunition  plants. 
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Two  years  was  the  longest  period  for  which  a  commercial  system 
located  among  the  respondents  was  dormant.  The  system  was  stored  in 
a  heated  warehouse  and  was  covered  with  plastic  bags.  Reactivation 
problems  were  minimal. 

There  were  no  indications  of  humidity  problems  either. 

Other  users  had  worse  reactivation  experiences  with  less  dormant 
storage.  Poor  or  sloppy  layaway  procedures  and  conditions  were  blamed 
for  the  problems,  rather  than  the  length  of  time  or  the  equipment. 

These  conclusions  were  drawn  from  the  respondents  to  this 
study. 

Vendors  of  £PC  systems  design  the  systems  to  operate  24  hours  per  day,  365 
days  per  year  and  rarely  give  much  thought  to  the  problems  of  long-term 
storage. 

Spare  parts  are  stored  for  long  periods  of  time,  but  vendors  warrant  the 
parts  for  only  three  to  five  years  so  that  the  concern  over  storage 
extends  only  that  far. 

Both  users  and  vendors  suggested  leaving  the  EPC  system  on  during  dormant 
periods. 

This  is  a  "play-it-safe"  approach  due  to  lack  of  knowledge  concerning 
the  effects  of  storage  on  the  components. 

Research  in  MMT  4322  should  be  referred  to,  as  it  will  shed  new  iight  on  this 
issue. 
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B.  RECOMMENDATIONS 


1.  ARRCOM  SYSTEM  DEVELOPMENT  CENTER 

•  ARRCOM  should  establish  or  support  a  technical  group  to  provide  long-range 
planning  for  the  annmunition  plant  modernization  program,  to  coordinate  all 
EPC  systems  activities  between  operating  contractors,  and  to  moderate  the 
use  of  outside  contractors  in  all  phases  of  an  EPC  systems  installation. 

This  group  should  establish  and  maintain  control  over: 

An  extensive  documentation  center. 

A  spare  parts  management  program. 

A  skilled  personnel  tracking  program. 
Operating  contractors  should  be  represented  in  this  group. 

2.  PROCUREMENT  PROCESS 

•  EPC  system  layaway  considerations  should  be  incorporated  into  the  initial  bid 
solicitation. 

Responses  to  the  solicitation  should  be  evaluated  for  system  lifetime 
costs,  including  spares,  maintenance,  power  consumption  under  various 
operational  modes,  and  for  layaway/reoctivation  costs. 

These  should  be  evaluated  during  development  and  modified  if  neces- 
sary. 

Component  degradation  data  should  be  requested  from  all  vendors  based 
upon  postulated  storage  conditions. 
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inherent  reliability  calculations  should  be  made. 

Spare  parts  and  testing  equipment  for  the  spares  should  be  funded  with  the 
initial  procurement. 

Complete  and  up-to-date  documentation,  below  the  board  level,  should  be 
requested  in  each  solicitation.  Documentation  should  be  revised  to  show  "as 
built"  condition  when  the  plant  is  proven  out. 

Software  documents  for  both  source  and  machine  language  codes  should 
also  be  included. 

STANDARDIZATION 

Very  definite  attempts  should  be  made  to  standardize  the  systems  used  in  the 
various  EPC  projects. 

As  much  standardization  as  possible  would  reduce  the  number  and  types 
of  spares  carried  and: 

Maximize  the  usefulness  of  the  available  skilled  personnel. 

*         Provide  leverage  with  the  vendor  for  support. 

Reduce  documentation  proliferation. 

Such  standardization  requires  choosing  vendors  with  broad  compatible 
product  lines  that  will  be  effective  in  handling  the  requirements  of  the 
majority  of  the  ammunition  plants. 

Broad  standardization  of  equipment  would  allow  the  operating  contractors 
more  responsibility  for  the  systems  and  reduce  the  number  of  vendors  involved 
in  the  entire  process. 
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4.       DORMANCY  TESTING 


•  ARRADCOM  should  continue  to  broaden  its  testing  program  which  determines 
the  effects  of  long-term  dormant  storage  on  EPC  systems. 

Vendors  have  no  interest  in  such  testing  unless  fully  funded. 

Users  have  no  need  generally.  Some  users  are  waiting  for  the 
recommendations  of  this  report  and  other  ARRADCOM  reports  before 
doing  anything  with  their  systems. 

•  The  tests  should  be  broadened  to  include: 

The  use  of  anticorrosive  vapors  for  protection. 
Various  types  of  environmental  control. 

The  use  of  remote  diagnostic  centers  for  system  reactivation. 

5.  SYSTEM  LAYAWAY 

•  Based  on  survey,  and  discussed  later  in  the  report,  systems  should  be  laid  away 
for  two  years  between  reactivations. 

These  systems  should  be  protected  with  plastic  bags  as  discussed  later 
for  dust  control  and  should  remain  in  a  computer  type  of  controlled 
environment. 

Spare  parts  must  be  treated  in  the  same  manner  as  the  systems. 

6.  THIRD-PARTY  MAINTENANCE  COMPANIES 

•  The  use  of  third-party  maintenance  companies  should  be  considered  for 
maintaining  the  EPC  systems  over  extended  periods  of  time. 


-  15- 


INPUT 


These  companies  will  maintain  the  entire  automated  line. 

They  will  keep  a  cadre  of  personnel  trained  in  the  necessary  specific 
systems  available. 

Many  maintenance  service  companies  will  manufacture  or  obtain 
obsolete  parts  for  the  older  systems. 

•        Third-party  maintenance  companies  are  not  interested  In  single  plant  con- 
tracts, preferring  to  handle  multiple  plants. 
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ill    INDUSTRY  TRENDS 
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ill       INDUSTRY  TRENDS 


A.       INTRODUCTION  TO  PROCESS  CONTROL 


I.  DESCRIPTION 

•  Process  control  is  defined  as  the  use  of  nnonitoring  instruments  and  control 
devices  to  measure  and  effect  changes  on  one  or  more  variables  to  ensure 
safe,  timely  and  efficient  operation  of  machines  and  other  equipment  required 
in  a  manufacturing  process. 

While  processes  may  be  categorized  by  the  type  of  materials  which  are 
to  be  effected  (i.e.,  fluids,  bulk  solids,  sheeted  and/or  webbed  m.ateri- 
als,  and  discrete  pieces),  the  concept  of  control  and  its  objectives 
remain  much  the  seme  for  all  types  of  materials. 

•  Electronic  process  control  refers  to  the  use  of  one  or  more  computers  to  aid  or 
actually  perform  the  control  functions  required  to  effect  change  in  a  manufac- 
turing process. 

Control  is  the  decision-making  operation  which  compares  the  measured 
state  of  a  process  with  the  desired  conditions  and  decides  how  the 
variables  should  be  manipulated. 
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In  electronic  process  control,  this  decision-making  operation  is  effected 
by  one  or  more  computers.  The  manipulation  of  the  variables  may  also 
be  effected  by  a  computer  either  in  a  fully  off-line  situation  (indirectly 
where  control  problems  for  a  particular  process  are  worked  out  and  the 
information  used  to  design  better  control  system  configurations  or  to 
provide  optimum  controller  settings)  or  in  a  direct  digital  control  (DDC) 
system  where  a  digital  computer  assumes  all  control  computations 
required  and  generates  signals  to  the  final  controlling  elements. 

•  Current  process  control  systems  operate  (with  computer  and/or  human  inter- 
face) in  a  real-time  mode  In  either  closed-loop  or  open-loop  configurations. 

Closed-loop  systems  generally  exclude  the  human  interface,  but  may  be 
only  a  smaller  part  of  a  larger  open-loop  system.  Closed-loop  systems 
and,  more  and  more,  DDC  systems  operate  in  real-time.  They  are 
generally  superior  in  processes  requiring  reaction  times  of  less  than  one 
minute  on  a  repetitive  basis  and  in  processes  regarded  as  unsafe  for 
human  intervention. 

Open-loop  process  control  systems  require  an  interface  with  one  or 
more  skilled  human  specialists  who  are  responsible  primarily  for 
monitoring  displays  of  feedback  control  systems'  parameter  data, 
implementing  the  computer  controlled  decision  matrix  algorithms,  and 
then  activating  operator  controls  in  accordance  with  computer-made 
decisions  to  effect  necessary  change  to  the  process. 

Another  term  often  used  in  referring  to  open-loop  control  systems  is 
the  operator  guide  system.  In  this  s/stem,  a  skilled  hum.an  specialist 
sets  knobs,  dials,  or  switches  to  control  the  operation  of  the  plant  based 
on  the  information  supplied  by  the  control  computer. 

•  Tne  essence  of  control  theory  is  based  on  mathematical  models  developed 
empirically  to  satisfy  the  neea  for  higher  levels  of  safety,  increased  levels  of 
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accuracy  in  process  tolerances,  better  economies  of  process  or  production 
scale,  and  increased  levels  of  system  performance. 

Since  an  increase  in  productivity,  safety  levels,  etc.  in  a  single  stage  of 
a  process  can  have  adverse  effects  on  any  other  stage  of  the  total 
process  system,  it  becomes  necessary  in  any  evaluation  to  analyze  these 
effects. 

•  The  analysis  of  the  total  process  control  system  can  be  carried  out  with 
mathematical  methods,  experimental  observation,  simulation,  or  a  combina- 
tion of  these  methods. 

Of  the  three,  mathematical  methods  are  the  least  expensive  but  are 
limited  to  rather  simple  processes.  As  the  complexity  of  the  system 
grows,  the  need  for  more  sophisticated  analysis  tools  becomes  neces- 
sary. 

Experimental  observation  is  the  most  expensive  method  since  it 
requires  the  installation  of  the  actual  control  equipment  to  be  used  in 
later  full-scale  operations. 

Today,  simulation,  where  the  entire  process  control  system  is  dupli- 
cated in  computer  software,  plays  a  critical  role  in  the  development, 
testing  and  implementation  of  process  control  systems.  It  is  capable  of 
handling  complex  systems  based  on  empirical  data  being  mathematic- 
ally and  graphically  manipulated  without  the  requirement  of  procuring 
and  installing  process  control  equipment  first. 

2.        BRIEF  HISTORY  AND  TRENDS 

•  Before  tne  development  (in  the  late  1920s)  of  the  pneumatic  controller  with  its 
proportional  and  integral  (reset)  modes  that  were  field  tunable  and  the 
proportional-derivative  (PID)  three-term  controller  (in  1942),  most  process 
control  had  to  be  impiemented  locally  and  manually. 


-  19  - 


INPUT 


The  pneumatic  transmitters  that  were  developed  in  the  mid- 1 930s  led 
to  the  first  use  of  centralized  control  rooms  with  large  control  panels  in 
the  early  1940s. 

The  development  of  the  PID  three-term  controller  in  1  942  also  marked 
a  departure  from  the  theoretically  based  development  of  control 
systems.  It  marked  the  beginning  of  control  system  developments  in 
plants. 

While  most  controllers  through  the  mid-1950s  were  pneumatic,  the  first 
electronic  controllers  were  marketed  commercially  as  early  as  1951. 

DC  electronic  controllers,  based  on  transistor-operational  amplifiers, 
started  to  gain  market  share  significantly  in  1960  but  did  not  surpass 
the  market  for  pneumatic  controllers  until  1970. 

Programmable  controllers  (PCs)  first  became  available  in  1970. 

The  first  electronic  analog  computer  was  developed  at  the  Foxboro  Company 
of  Foxboro,  Mass.,  in  1938.  The  first  installation  of  a  computer-controlled 
system  occurred  in  1959  when  the  industry  buzz-word  was  "adaptive  control." 

Some  limited  closed-loop  computer  control  was  used  as  early  as  1955  in 
aircraft  flight  control  decisions,  in  1958  in  monitoring  in  electric 
utilities,  in  1959  in  supervisory  control  in  a  refinery,  and  in  1960  for 
supervisory  control  in  a  chemical  plant. 

Digital  computers  were  first  used  in  the  process  control  field  in  1955  to 
simulate  a  process  control  system  configuration.  The  first  use  of  DDC  was 
around  i  962. 

Centralized  digital  control  was  the  theme  in  industry  in  the  period  from 
1962  through  about  1974. 
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•  Initially,  the  central  computers  were  large  mainframes;  but  as  minicomputer 
reliability  grew,  they  rapidly  became  the  heart  of  the  electronic  process 
control  system. 

As  minicomputer  power  grew,  hierarchies  of  minicomputers  were 
constructed  with  the  aim  of  total  plant  control,  as  shown  in  Exhibit 
lil-l. 

One  respondent  in  INPUT'S  survey  is  planning  for  30,000  control  loops. 
At  the  current  stage,  8,000  loops  are  under  computer  control. 

•  As  integrated  circuit  technology  developed  and  the  microcomputer  came  into 
being,  these  technologies  moved  into  the  process  control  field. 

Programmable  logic  controllers  (PLC)  for  single  loop  control  were 
placed  in  the  field  or  at  the  machining  location  to  control  individual 
operations. 

The  name  has  now  changed  to  programmable  controllers,  and  they 
contain  all  the  elements  of  full  computers.  They  can  control  from  one 
to  about  2,000  loops  and  are  rugged  enough  to  operate  in  a  normal 
factory  environment  or  under  adverse  field  conditions. 

•  This  use  of  PCs  is  called  distributed  control,  and  the  term  data  highway  is  in 
vogue  describing  the  information  path  among  the  computers  and  programmable 
controllers. 

This  trend  toward  distributed  or  decentralized  control  will  intensify 
during  the  eighties  because  several  advantages  accrue  from  a  distrib- 
uted system  design. 

Portions  of  the  process  can  be  automated  quickly,  providing  an 
early  testing  of  the  approach  and  perhaps  some  productivity 
gains. 
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EXHIBIT 


HIERARCHICAL  ORGANIZATION  OF  AN 
ADVANCED  PROCESS  CONTROL  SYSTEM 
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A  portion  of  the  system  may  fail  without  causing  the  production 
line  to  stop. 


No  single  control  point  is  extremely  critical  so  the  costs  of 
achieving  reliability  through  redundancy  are  reduced.  It  is  less 
expensive  to  add  redundant  equipment  in  a  few  critical  areas 
than  it  is  to  provide  duplicate  central  control  computers. 

•  The  evolution  of  digital  computer  control  systems  over  the  past  15  years  is 
shown  in  Exhibit  III-2. 

•  While  the  eighties  will  emphasize  practical  distributed  control  for  increased 
systems  reliability,  the  R&D  efforts  will  be  on  developing  sensors  that  are 
more  accurate  and  able  to  measure  more  variables. 

Distributed  control  will  remain  the  predominant  mode  during  the  1980s. 

•  The  early  major  industrial  users  of  electronic  process  control  (EPC)  systems  in 
the  first  half  of  the  1960s  were  the  large  process  industries. 

Chemicals. 

Utilities. 

Petroleum  production  and  refining. 

Steel  production. 

Pipelines. 

•  More  recent  process  industries  that  have  become  major  users  of  computers  to 
control  manufacturing  are: 

Pulp  and  paper  production. 
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EXHIBIT  111-2 
EVOLUTION  OF  DIRECT  DIGITAL  CONTROL  SYSTEMS 
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Food  and  beverage. 


Nuclear  power  generation. 

Computers  have  been  involved  in  aspects  of  the  discrete  manufacturing 
process  since  the  early  1960s,  but  they  have  only  recently  been  involved  in 
controlling  those  processes.  The  major  discrete  manufacturing  industries  that 
have  pioneered  the  use  of  computer-controlled  production  lines  or  local 
machining  operations  are: 

Automotive. 

Aerospace. 

Machine  tools. 

The  increasing  use  of  microcomputers  and  programmable  controllers  will  allow 
manufacturing  operations  of  many  sizes  to  automate  portions  of  the  produc- 
tion process  efficiently. 

The  greatest  growth  during  the  eighties  in  the  process  control  industry 
will  take  place  in  the  discrete  manufacturing  sector. 

MARKET  SIZE 

There  is  no  need  in  this  study  to  do  a  definitive  sizing  of  the  process  control 
market. 

However,  some  feel  for  the  size  of  the  market  is  useful  in  any  industry 
description. 

The  process  control  market  growth  figures  shown  in  Exhibit  111-3  have  been 
adapted  from  those  published  by  the  instrument  Society  of  America  (ISA). 
They  include  all  aspects  of  the  control  industry  such  as: 
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PROJECTED  TOTAL  SPENDING  FOR  PROCESS  CONTROL  EQUIPMENT, 

1970-1990 


3  i  I  \  \  \  '  I  \  \  I  I 

1970     1972     1974     1976     1978     1980     1982     1984      1985      1988  1990 

COMPOUND  ANNUAL  GROWTH  IS  <  14%  /  YEAR 
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Analytical  instruments. 

Computers  and  computer  periplnerals. 

Controllers,  counters  and  timers. 

Data  display  systems. 

Final  control  elements. 

Packaged  control  systems. 

Sensors,  transducers  and  transmitters. 

Signal  conditioners. 

•  The  electronic  process  control  system  forecast  shown  in  Exhibit  111-4  is  based 
upon  projections  published  in  the  January  13,  1981,  issue  of  Electronics 
Magazine  and,  while  it  represents  only  a  portion  of  the  total  spending,  it  is 
still  considerably  more  conservative  than  the  ISA-based  forecast. 

The  compound  growth  rate  for  the  industry  has  been  between  10%  and 
15%  per  year  over  the  last  10  years.  The  programmable  controller 
portion  will  probably  grow  at  20%  to  22%  per  year  during  the  next  10 
years,  but  the  overall  industry  growth  will  remain  in  the  10-15%  range. 

INPUT  believes  these  estimates  are  adjusted  to  exclude  inflationary 
factors  so  that  they  reflect  real  growth. 

B.       MAiNTENANCE  TRENDS  IN  THE  COMPUTER  INDUSTRY 


•         The  Army  should  be  aware  of  the  major  maintenance  trends  in  the  industry. 
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EXHIBIT  III-4 


PROCESS  CONTROL  COMPUTER  SYSTEM  FORECAST 


($  MILLIONS) 


SYSTEM 

1979 

1980 

1981 

1984 

DIGITAL 

$190 

$212 

$232 

$304 

ANALOG 

40 

42 

46 

55 

TOTAL 

$230 

$254 

$278 

$359 

SOURCE:  ELECTRONICS  MAGAZINE  JANUARY  13,  1981 
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These  trends  can  impact  both  the  availability  and  quality  of  mainte- 
nance at  ammunition  plants. 

Some  methods  and  procedures  being  developed  in  the  computer  industry 
may  be  useful  to  the  Army. 

The  factors  discussed  in  this  section  should  be  taken  into  consideration 
in  future  equipment  and  maintenance  specifications  and  procurements. 

•  This  chapter  covers  a  number  of  important  trends  in  the  computer  industry's 
maintenance  functions.  INPUT  has  attempted  to  highlight  those  areas  that 
could  be  useful  to  the  Army. 

INPUT  has  a  Field  Service  Program  which  prepares  eight  to  ten  reports 
annually  on  the  subject  of  maintenance. 

Clients  benefit  from  studies  on  management,  control,  trends,  service 
levels,  and  other  current  topics  on  maintenance. 

This  service  may  be  of  value  to  ARRADCOM. 

I.        DESIGN  FOR  MAINTAINABILITY 

a.        Equipment  Design  for  Reliability  and  Serviceability 

•  Beginning  in  the  mid-1970s,  a  few  equipment  designers  began  to  place 
increasing  importance  on  reliability  and  serviceability  as  a  design  criteria. 

Although  the  importance  of  this  criteria  varies  substantially  from 
manufacturer  to  manufacturer,  there  is  no  question  that  this  is  an 
industry  trend. 

•  Design  for  reliability  and  serviceability  is  taking  many  forms.  Some  of  these 
are: 
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Redundant  circuitry. 


Use  of  components  operating  well  below  design  limits. 


Improved  materials  used  on  components  that  have  a  wear  factor  with 
which  to  contend  (e.g.,  magnetic  heads). 

Built-in  diagnostics. 

Built-in  instrumentation  and/or  separate  but  specialized  instrumen- 
tation. 


Redundant  functional  modules. 


Modular  design. 


No  one  manufacturer  currently  uses  all  of  these  design  approaches,  but  most 
either  nave  or  plan  to  have  several  of  these  features  designed  into  their 
product  lines. 

An  outstanding  example  is  that  of  Tandem,  a  California-based  computer 
manufacturer,  whose  standard  product  is  a  dual  processor  minicomputer 
designed  so  that  one  processor  automatically  takes  over  from  the  other 
should  it  fail.  Failure  detection  and  switching  are  built  in  and  are 
automatic. 

IBM  uses  triple  redundant  circuitry  in  many  of  its  newer  products.  As 
the  system  runs,  the  outputs  from  three  identical  circuits  (which 
operate  in  parallel)  are  compared.  If  the  output  of  one  circuit  differs 
from  the  other  two,  the  system  automatically  chooses  one  of  the 
circuits  in  the  majority. 
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b.  Testing 


•  The  increased  availability  and  lower  cost  of  intelligence  has  spurred  the 
development  of  two  new  prinnary  instrunnentation  approaches  to  maintenance 
that  will  be  increasingly  evident  in  the  next  few  years: 

Systems  designed  to  accommodate  remote  diagnostics. 

Standalone  maintenance  "computers." 

•  Remote  diagnostics  (which  are  discussed  in  more  depth  in  Section  B.3  of  this 
chapter)  are  particularly  suited  to  installations  where  all  the  equipment  has 
been  provided  by  a  single  vendor. 

•  There  are  two  approaches  to  testing  which,  alone  and  together,  constitute  an 
industry  trend.  These  are: 

Built-in  testers  (instruments). 

Use  of  microdiagnostics. 

•  Microdiagnostics  can  be  hardwired  or  loadable.  They  can  be  tailored  to  fit 
differing  requirements.  For  example,  a  simple  test  designed  for  user  operation 
may  be  provided,  while  complex  tests  designed  to  be  used  by  skilled  service 
personnel  can  be  run  when  the  occasion  calls  for  it. 

•  Microdiagnostics  will  be  used  extensively  on-site  in  multivendor  installations. 

•  There  is  also  a  trend  toward  incorporation  of  controllers  and  other  decision- 
making devices  into  interfaces  permitting  direct  device  attachment  to  buses 
or  channels. 

These  "smart  front  ends"  provide  an  opportunity  to  build  in  diagnostic 
systems  capable  of  monitoring  and  self-testing  both  interfaces  and 
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peripheral  devices.  They  greatly  enhance  the  ability  of  service  people 
to  isolate  faults  in  multi-element  systems. 

PERSONNEL  TRENDS 

Demographic,  motivational  and  industry  growth  forces  have  combined  to 
create  a  substantial  shortage  of  skilled  service  personnel  in  the  computer 
industry. 

This  shortage,  presently  estimated  by  INPUT  at  25,000  people  in  the 
U.S.  alone,  will  continue  to  intensify  as  new  systems  and  equipment  are 
added  to  the  already  substantial  installation  base. 

The  computer  industry,  which  used  to  obtain  the  majority  of  its  skilled 
technicians  from  the  military  before  the  days  of  the  all-volunteer 
service,  has  simply  not  been  able  to  recruit  and  train  the  quantity  and 
quality  of  people  it  needs. 

Because  of  this  phenomenon,  the  computer  industry  is  being  forced  to  change 
the  methods  by  which  it  delivers  maintenance.  As  a  consequence,  the  skills 
mix  is  also  changing.  This  section  projects  the  evolution  of  the  "serviceperson 
of  the  future." 

3y  the  mid-1980s,  the  service  organization  will  evolve  into  a  three-tiered 
structure.  At  each  level,  different  types  of  people  will  be  required  to  meet 
the  service  demcnds  of  the  integrated  computer/office/communications 
systems  in  place  at  that  time. 

Within  this  hierarchy,  each  level  has  its  own  role  in  the  service  organization, 
performing  different  functions  and  interfacing  with  customers  in  different 
ways  and  often  at  different  levels  (i.e.,  the  skills,  training,  handling  and 
compensation  requirements  will  be  significantly  different  for  each  category). 


\ 
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•  Due  to  economic  necessity,  some  of  the  functional  requirements  of  the  three 
levels  may  overlap,  depending  on  customer  base,  product  line  density,  geo- 
graphic dispersion  of  specific  products  and  other  factors.  In  most  organiza- 
tions, however,  the  lines  will  be  distinctly  drawn. 

•  The  three  classes  of  service  personnel  and  their  distinguishing  characteristics 
are  described  as  follows: 

a.  Local  Field  Service  Technician  (LFST) 

•  LFSTs  exist  primarily  to  handle  routine  service  functions  such  as: 

"Cookbook"  preventive  maintenance  diagnostics  and  routines. 
Repair  at  the  module  or  unit  replacement  level. 
Cleaning  and  burnishing  of  mechanical  components. 

•  Skill  and  technical  training  requirements  for  LFSTs  are  relatively  minimal. 

b.  Local  Field  Support  Specialist  (LFSS) 

•  LFSSs  serve  as  backup  to  the  LFSTs,  taking  charge  of  problems  the  latter 
cannot  deal  with.  They  must  have  a  thorough  grounding  in  the  theory  of 
operations  of  the  systems  with  which  they  will  come  in  contact. 

•  Most  positions  will  require  the  equivalent  of  a  four-year  degree.  Most  will 
need  to  comprehend  both  software  and  communications. 

•  LFSSs  will  be  generalists  capable  of  making  decisions  on  nonroutine  problems 
as  they  come  across  a  broad  spectrum  of  products  and  systems. 
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c.        Central  Site  Support  Specialists  (CSSS) 

•  Central  site  support  specialists  are  resident  at  headquarters  or  regional 
locations  and  are  responsible  for  dealing  with  highly  technical  problems  beyond 
the  capabilities  of  field  personnel. 

•  CSSSs  are  specialists  in  every  sense  of  the  word.  They  have  in-depth 
knowledge  of  specific  products;  indeed,  their  knowledge  may  be  limited  to  an 
individual  subset  of  a  complex  system;  for  example,  the  central  logic  unit  or 
the  operating  system  software  package. 

•  The  change  in  the  distribution  of  skills  in  a  field  service  organization  is  shown 
graphically  in  Exhibit  111-5. 

3.        NEW  MAINTENANCE  TECHNIQUES 

•  This  section  addresses  four  "new"  maintenance  techniques  that  are  gaining 
rapid  acceptance  throughout  the  computer  industry: 

Remote  diagnostics. 

Self-maintenance  procedures. 

Depot  maintenance. 

System  support  centers. 

•  All  these  methods  could  play  an  important  role  in  the  achievement  of  the 
Army's  objective  of  high  long-term  reliability  under  dormant  conditions. 

a.        Remote  Diagnostics 

•  Remote  diagnostics  have  been  available  for  several  years,  but  vendors  are  just 
beginning  to  exploit  the  potential  of  these  techniques. 
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EXHIBIT  1II-5 
SHIFT  IN  DISTRIBUTION  OF  SKILL  LEVELS 
OF  FIELD  SERVICE  PERSONNEL 


1980 


ENTRY  LEVEL  FIELD  ENGINEER 


AVERAGE  FIELD  ENGINEER 


HIGHLY  SKILLED  FIELD 
ENGINEER 


SKILL  LEVEL- 
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Although  development  of  remote  diagnostics  is  being  driven  by  escalat- 
ing labor  costs  and  the  shortage  of  qualified  service  personnel,  vendors 
are  finding  that  they  can  actually  provide  better  service  at  less  cost 
with  remote  diagnostics. 

A  side  benefit  to  the  implementation  of  a  remote  diagnostic  system  is 
that  service  levels  tend  to  become  standardized  across  all  geographic 
regions. 

INPUT  recently,  as  part  of  its  on-going  research,  surveyed  the  U.S.  computer 
industry  and  found  that  only  10%  of  vendors  had  no  plans  for  remote 
diagnostics. 

Of  the  remaining  90%,  one-half  already  have  some  capability. 

The  other  half  had  plans  to  implement  remote  diagnostics  for  new 
products,  with  scattered  reports  of  retrofitting  this  capability  to  older 
equipment. 

In  the  technological  sense,  the  limiting  factor  is  not  In  the  robot  and 
communicctio.is  segment,  but  in  the  quality  and  reliability  of  the  diagnostics 
themselves. 

Remiote  access  and  control  of  operator  and  maintenance  consoles  is  a 
relatively  simple  electronic  achievement. 

Remote  control,  however,  will  not  improve  the  reliability  of  the  data 
reporting  capabilities. 

At  present  the  value  added  in  most  instances  is  the  quicker  introduction  of 
interpretive  capabilities  to  the  local  situation  via  remote  hook-up. 

One  company  reports  that  as  many  as  15%  of  the  calls  on  peripherals 
ore  now  avoidable  because  of  their  remote  verification  process  which 
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aids  the  dispatch  center  in  directing  users  to  call  their  mainframe 
maintenance  vendor  in  those  cases  where  the  fault  lies  with  the 
mainframe. 

•  Equipment  reliability  has  paradoxically  created  a  need  for  remote  diagnostics 
as  well  as  for  more  reliable  diagnostics  generally. 

The  evolution  of  reliability  in  equipment,  following  design  principles 
developed  for  aeronautics  and  space  programs,  and  employing  redun- 
dancy and  backup  systems,  has  proved  to  be  a  two-edged  sword  in 
maintenance.  . 

Experience  levels  and  learning  rates  of  field  engineers  have  tradition- 
ally depended  upon  positive  reinforcement  derived  from  trouble- 
shooting natural  failures. 

With  more  reliable  equipment,  therefore,  the  field  engineer  has  less 
practice  with  failures  and  can  lose  individual  proficiency. 

•  There  are  many  different  remote  diagnostic  schemes  in  place  and  under 
development.  For  the  purpose  of  this  report,  INPUT  has  selected  Digital 
Equipment  Corporation's  (DEC)  new  system  as  a  particularly  meaningful 
example,  since  it  is  designed  to  handle  the  class  of  systems  encountered  at  the 
Army's  munitions  plants. 

•  DEC'S  system  Is  entitled  "Remote  Computerized  Diagnostic"  (RCD).  It  is  used 
with  PDP  i  l/70-based  systems  and  the  VAX  I  1/780.  It  became  operational  in 
early  1980. 

•  Based  on  the  performance  figures  available  in  late  1980,  DEC's  reported 
results  are  extremely  significant. 


-  37  - 


iNPUT 


Thirty  percent  of  all  faults  are  solved  over  the  telephone  by  having  the 
on-site  operator  take  sonne  simple  corrective  action  without  any 
participation  by  a  skilled  service  person. 

RCD's  success  rate  in  isolating  the  failed  part  of  the  system  in  the 
remaining  70%  of  the  reported  faults  is  90%.  Service  technicians  are 
immediately  dispatchea  with  the  required  repair  parts  in  hand. 

•  One  of  the  principal  features  of  the  DEC  system  (and  others  like  it)  is 
centralized  fault  reporting,  enabling  the  automatic  recording  of  all  system 
failures  and  fixes. 

This  is  consistent  with  ARRADCOM's  approach  and  what  it  is  attempt- 
ing to  accomplish. 

Statistical  analysis  of  this  data  can  then  be  used  to  make  a  major 
contribution  to  reliability  improvement  programs,  which  are  of  partic- 
ular interest  to  the  Army. 

ARRADCOM  has  the  data  for  extensive  testing  and  improvements  over 
the  next  20  years. 

•  DEC'S  RCD  system  is  shown  schematically  in  Exhibit  111-6. 

The  nost  system,  resident  at  the  remote  diagnostic  center,  consists  of 
dual  PDP  ll/70s  interfaced  to  a  network  controlled  by  PDP  1  1 /04s. 
The  latter  provide  message  switching  and  auto-dial  facilities. 

The  user's  system  requires  a  special  console,  designated  the  KYI  1-RE, 
which  replaces  the  standard  operator  console. 

The  console,  which  contains  a  built-in  microprocessor,  has  its 
own  self-test  facility  so  that  the  operator  can  determine  if  The 
console  itself  has  a  fault. 
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VAX  systems  have  the  console's  special  facilities  built  in. 


An  on-line  disk  storage  unit  at  the  host  system  contains  data  on 
customers,  their  system's  performance  history,  and  their  machine 
configuration. 

Diagnostic  routines  are  maintained  at  the  customers'  sites  and  kept  on 
disk  and/or  tape. 

•         System  operation  procedures  for  the  DEC  RCD  system  are  shown  diagram- 
matically  in  Exhibit  III-7. 

When  a  fault  is  detected,  the  customer  initiates  the  procedure  by 
telephoning  the  "Service  Response  Group"  (SRG)  located  at  the  remote 
diagnostic  center. 

The  SRG  engineer  logs  the  call  and  discusses  the  fault  with  the 
customer.  If  he  determines  that  the  fault  Is  operator-induced  or  is  an 
environmental  problem,  the  customer  is  advised  accordingly.  If  it  is  a 
software  problem  which  the  SRG  engineer  is  not  able  to  resolve 
himself,  the  contact  will  be  passed  to  a  software  support  group. 

If  the  fault  is  determined  to  be  a  straightforward  peripheral  failure,  the 
SRG  passes  the  call  to  the  appropriate  local  service  branch  which 
dispatches  a  field  engineer  to  the  customer's  site. 

If  the  fault  is  thought  to  be  a  mainframe  or  system  problem,  remote 
diagnostics  will  be  used. 

To  operate  the  remote  diagnostic  system,  the  customer  first 
removes  his  storage  media  or  ensures  that  it  is  write-protected. 
He  then  switches  the  console  to  "remote,"  which  effectively 
passes  control  of  the  system  to  the  remote  diagnostic  center. 
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EXHIBIT  111-7 


DEC  SYSTEM  OPERATION  AND  PROCEDURE 


REMOTE 
DIAGNOSTICS  TEAM 
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The  local  service  branch  is  immediately  advised  to  enable  pre- 
scheduling  of  a  possible  visit  by  a  field  engineer. 

Diagnostic  tests  are  then  run  and  monitored  by  the  SRG  engineer 
until  a  fault  or  test  failure  is  encountered.  Test  results  and  data 
are  displayed  on  a  DEC  writer  at  the  remote  diagnostic  center. 

A  local  service  branch  is  then  advised  to  dispatch  a  field 
engineer  who  will  be  armed  with  the  parts  needed  to  repair  the 
fault. 

Once  the  problem  is  resolved  by  repair,  the  field  engineer 
informs  the  SRG  engineer  who  then  reruns  the  appropriate 
diagnostics  to  prove  that  the  repair  has  been  effected.  Control 
is  then  returned  to  the  customer. 

One  of  the  more  interesting  features  of  the  DEC  system  is  that  the  SRG  can 
run  parts  of  the  preventive  miointenance  test  schedule.  This  is  particularly 
attractive  because  these  tests  can  be  executed  during  non-prime  shift  hours. 

On  the  VAX  I  1/780,  when  an  operator  suspects  or  detects  a  hardware  error, 
he: 

Mounts  a  diagnostic  disk  pack. 
Loads  a  diagnostic  floppy  disk  in  the  console. 
Selects  "remote  diagnostic"  on  the  console  switch. 
Calls  the  diagnostic  center. 
Once  he  has  done  this,  no  other  activity  is  required  from  the  site. 


-  42  - 


iNPL 


•  The   diagnostic  center   runs  the    11/780  automated  diagnostic  package  or 
operates  the  diagnostic  console  manually. 

•  If  a  problem  is  found,  then  a  field  engineer  is  sent  with  the  required  equipment 
and  replacement  modules. 

•  Using  this  approach,  reactivation  could  be  effected  under  the  control  of  the 
remote  diagnostics  center. 

b.        Self-Maintenance  Procedures 

•  In  an  effort  to  cover  the  shortfall  of  maintenance  personnel,  some  vendors  are 
encouraging  the  direct  participation  of  their  users  in  the  maintenance  process. 

•  Most  vendors  view  self-maintenance  as  one  of  a  series  of  elements  in  a 
complete  maintenance  strategy.  ' 

Other  elements  include,  for  example,  the  use  of  system  support  centers, 
modular  equipment  design  and  remote  diagnostics. 

•  Self-maintenance  programs  are  not  simple  to  implement.    Vendors  usually 
must  provide: 

Special  diagnostic  routines  designed  to  be  run  by  the  unskilled. 
Thorough  documentation. 
User  training  programs. 

Special  product  design  and  packaging,  in  some  cases. 

•  Nonetheless,  self-maintenance  programs  are  beginning  to  be  introduced  and 
INPUT  believes  they  will  be  a  growing  trend. 
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IBM  is  experimenting  in  this  approach  with  its  new  3101  display 
terminal  and  3102  thermal  printer.  The  self-maintenance  concept  was 
engineered  into  these  products  from  the  very  beginning.  In  fact,  IBM 
offers  no  alternative  service  policy  on  these  products. 

•  A  major  driving  force  to  self-maintenance  is  the  growing  use  of  distributed 
processing. 

-  By   1982,  INPUT  estimates  that  30%  of  the  total  installed  base  of 
computer  equipment  will  be  operating  in  a  distributed  environment. 

Distributed  systems  are  particularly  suited  for  user  self-maintenance 
because  communications  facilities  are  already  there,  the  equipment  at 
a  given  site  is  relatively  simpler  than  large  systems,  and  the  expense  of 
on-site  service  coverage  is  usually  not  justified. 

•  Today,  self-maintenance  is  not  common.  Most  self-maintenance  functions  are 
limited  to  running  diagnostics  using  cookbook  procedures  that  can  be  learned 
by  anyone. 

-  In  a  few  cases,  users  are  performing  board  or  module  replacement. 

In  even  fewer  cases,  users  are  taking  prime  responsibility  for  all 
maintenance  functions.  This  kind  of  activity  tends  to  be  restricted  to 
high-technology  companies  who  already  have  employees  with  the  right 
set  of  skills.  For  example,  it  Is  fairly  common  for  an  electronics 
equipment  firm  to  maintain  test  equipment  manufactured  by  someone 
else. 

c.        Depot  Maintenance 

•  Depot  maintenance,  often  provided  together  with  user  self-diagnostics,  is  a 
third  maintenance  technique  grov/ing  in  importance. 
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Few  process  control  vendors  supply  such  a  service  today,  but  INPUT 
sees  this  as  a  trend  with  computer  manufacturers  which  may  spread  to 
process  control  vendors. 

Depot  maintenance  is  practical  only  for  equipment  that  can  easily  be 
packed  and  shipped.  With  the  physical  size  of  electronics  packaging 
decreasing,  more  equipment  will  be  suited  for  depot  maintenance  in  the 
future. 

•  Depot  maintenance  offers  two  significant  benefits  to  the  customer: 

It  is  usually  the  least  expensive  procedure. 

The  skills  and  equipment  (both  for  testing  and  repair)  found  at  depot 
maintenance  centers  are  of  the  highest  quality. 

Devices  sent  to  repair  depots  usually  go  through  an  exhaustive, 
complete  test  procedure  rather  than  having  attention  focused  only  on 
the  reported  fault  as  is  usually  the  case  with  on-site  maintenance. 
Thus,  faults  other  than  the  one  that  caused  the  system  to  be  sent  back 
to  the  depot  are  detected  and  fixed,  usually  at  no  additional  expense  to 
the  customer. 

d.        System  Support  Centers 

•  The  system  support  center  is  the  fourth  maintenance  technique  to  gain  new 
popularity.  As  it  is  being  implemented  today,  the  concept  is  simple:  take  the 
best  expertise,  equip  it  with  the  most  sophisticated  diagnostic  tools  and  aids, 
and  put  these  elements  together  in  a  single  location  accessible  to  all 
customers  by  both  voice  and  data  telecommunications  facilities. 

Although  a  few  relatively  small  firms  have  had  established  support 
centers  for  more  than  ten  years,  only  recently  have  the  largest 
companies  (most  notably  IBM)  implemented  them. 
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By  the  end  of  the  1980s,  INPUT  believes  there  will  be  almost  total  conversion 
of  first-line  user  support  from  the  field  to  the  support  center. 

Coupled  with  this  shift,  the  maintenance  function  itself  will  become 
95%  logistic  and  only  5%  diagnostic  because  of  new  diagnostic  aids, 
both  local  and  remote.  This  is  almost  a  complete  reversal  of  the 
situation  as  it  existed  in  the  late  1970s  when  most  maintenance 
resources  were  spent  on  diagnosis  rather  than  sparing  and  repair. 

This  will  be  important  to  ARRADCOM. 

The  distinguishing  characteristics  of  support  centers  as  they  are  evolving  are: 

Regional  or  national  support  centers  staffed  with  dedicated  support 
specialists. 

.  \ 

Procedural  requirements  for  field  personnel  to  attempt  diagnosis 
through  telephone  assistance  before  dispatching  specialists  to  provide 
physical  assistance. 

Centralization  of  data  on  symptoms  and  fixes. 

Higher  level  functional  authority  vested  in  technical  support  staff 
management. 

Formal  communications  linkage  between  support  centers  and  the  plants 
which  make  the  equipment  or  design  the  software. 

Integration  of  technical  support  with  other  support  functions  in 
diagnosis. 

Physical  planning  supocrt. 

Configuration/systems  assistance. 
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Software  support. 


Field  education. 
Engineering  change  control. 

"Alert"  system  and  status  monitoring  for  management. 
Asset  controls,  tools  and  test  equipment. 
Product  performance  tracking. 
Maintenance-level  component  failure  analysis. 
Technical  information  dissemination  control. 

•  A  typical  support  center  hierarchy  is  shown  in  Exhibit  111-8. 

The  procedure  looks  and  is  relatively  simple  to  effect. 

•  An  interesting  example  of  what  one  company  plans  to  achieve  with  its  support 
center  operations  in  the  software  area  is  illustrated  in  Exhibit  III-9.  Although 
this  is  only  a  model,  not  actual  results  (it's  too  early  to  tell),  the  chart  clearly 
shows  a  dramatic  potential  for  improvement. 

4.        MAINTENANCE  COST  TRENDS 

•  The  cost  of  maintenance  is  becoming  an  increasingly  higher  percentage  of 
computer  users'  expenditures  over  a  seven-year  operating  life  cycle  of  a 
system.  This  fact  is  illustrated  in  Exhibit  111-10  which  shows  the  ratio  between 
hardware  and  service  costs  over  a  typical  seven-year  product  operating  life 
cycle. 
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EXHIBIT  111-10 


TOTAL  USER  SYSTEM  EXPENDITURES 
OVER  A  SEVEN-YEAR  LIFE  CYCLE 


MAINTENANCE 
3  EQUIPMENT  PURCHASE 


NOTE:  CHART  ASSUMES  IN  YEAR  1  ^HAT  MAINTENANCE  IS  10%  OF 
PURCHASE  PRICE 


-  50- 


The  chart  assumes  current  annual  maintenance  charges  are  10%  of 
equipment  purchase  price,  which  is  typical  of  minicomputer  systems. 

Clearly,  users  are  making  a  financial  commitment  to  service  as  large  as 
or  greater  than  their  commitment  to  hardware  as  the  relative  price  of 
maintenance  increases. 

•  It  is  true  that  the  costs  of  maintaining  newer  systems  built  with  state-of-the- 
art  technology  are  lower  relative  to  older  systems  of  equivalent  power. 

Rarely,  however,  are  old  systems  replaced  with  ones  of  equivalent 
power,  so  the  absolute  cost  of  maintenance  to  the  user  in  most  cases  is 
not  diminished.  In  fact,  it  is  often  higher  because  of  increased 
complexity  due,  for  example,  to  the  addition  of  telecommunications 
capability. 

•  The  information  processing  industry  has  been  dominated  by  IBM  computers, 
Xerox  office  products,  and  AT&T  communications.  All  three  originally  took  a 
bundled  approach  to  service. 

Bundled  means  that  the  service  charges  were  included  in  the  products' 
rental  cost. 

Led  by  IBM  which  is  generally  separating  service  from  hardware,  the 
industry  is  seeing  the  bundling  constraints  crumble. 

•  Because  computer  systems  have  become  more  complex  and  at  the  same  time 
more  important  to  an  institution's  ability  to  function,  the  sensitivity  to  a 
system  being  "down"  is  becoming  increasingly  intensified. 

In  this  environment,  service  becomes  an  imperative  that  can  be 
justified  at  nearly  any  price. 
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MAINTENANCE  CONTRACT  TRENDS 


INPUT'S  research  shows  that  computer  users  elect  to  adopt  fixed-fee  mainte- 
nance contracts  over  time  and  materials  (T&M)  by  a  better  than  20:1  ratio. 
However,  there  is  a  trend  toward  reducing  base  coverage  (say  to  a  single  shift) 
and  purchasing  additional  spot  coverage  as  needed  on  a  T&M  basis. 

Computer  services  companies  will  provide  both  fixed  fee  and  T&M 
contract  alternatives  to  users. 

Several  emerging  trends  are  beginning  to  influence  the  composition  of  service 
agreements: 

increased  customer  involvement  in  diagnostic  and  repair  procedures. 

The  shrinking  size  of  equipment  which  makes  depot  maintenance 
practical. 

More  widespread  use  of  remote  diagnostics  and  system  support  centers. 

Due  to  these  influences,  some  important  changes  in  service  contracts  will  be 
made. 

On  the  one  hand,  vendors  will  provide  much  more  flexibility  to  meet 
individual  customer's  requirements. 

On  the  other  hand,  some  products  will  be  offered  on  a  "take-it-or- 
leave-it"  basis,  such  as  depot  maintenance  on  the  IBM  3101  terminal. 

A  choice  of  standard  plans  will  be  available  for  the  smaller  user. 

INPUT  believes  that  within  ten  years  nearly  all  post-warranty  service  will  be 
unbundled,  reflecting  general  recognition  of  service  as  a  product. 


-  52- 


INPU 


Unbundling  means  that  service  will  be  priced  separately  for  each  piece 
of  equipment. 

Although  it  is  not  widely  used  today,  one  is  beginning  to  see  experi- 
ments by  industry  leaders,  particularly  IBM,  that  point  to  much  broader 
use  in  the  future. 

•  Hardware  maintenance  costs  (in  relation  to  the  installed  base)  will  continue  to 
increase  over  the  next  three  to  four  years  and  may  begin  to  reduce  after  that 
time  as  hardware  becomes  more  reliable. 

However,  INPUT  believes  that  service  costs  for  maintenance  will 
continue  to  increase  for  the  foreseeable  future. 

The  increasing  complexity  of  distributed  systems  and  attendant 
software  will  add  more  to  software  maintenance  costs  than  cost- 
saving  devices  such  as  systems  support  centers  will  deduct. 

Thus,  the  total  cost  of  service  as  a  percentage  of  income  derived 
from  the  sale  of  a  system  is  expected  to  increase. 

•  In  summary,  the  general  computer  industry  is  definitely  moving  toward 
treatment  of  service  as  a  separate  product  which  will  increasingly  be  procured 
on  a  "pay  for  what  you  get"  and  "pay  as  you  go"  basis. 

6.        THIRD-PARTY  MAINTENANCE  (TPM) 

•  There  are  two  types  of  TPM  companies  in  the  computer  industry. 

TPM  firms  who  rely  on  contracts  to  provide  maintenance  on  obsolete 
equipment  and/or  mixed  vendor  installations  and  who  exist  largely 
because  the  manufacturers  want  them  to  exist. 

Their  prime  competitive  edge  is  price. 
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The  equipment  suppliers  could  drive  them  out  of  business  if  they 
choose  to  do  so,  simply  by  making  it  difficult  to  obtain  spare 
parts,  or  by  reducing  maintenance  prices. 

Another  way  to  preclude  end  user  TPM  is  to  prevent  the  TPM 
vendor  from  being  able  to  maintain  software. 

.         INPUT  uses  the  term  end  user  TPM  to  describe  this  business. 

The  second  class  of  TPM  business  involves  the  establishment  of  long- 
term  contracts  between  a  TPM  firm  and  a  manufacturer  wherein  the 
TPM  firm  agrees  to  handle  all  the  maintenance  of  a  product  or  product 
line  within  a  prescribed  geographic  territory  (which  may  be  national  or 
even  international). 

The  TPM  vendor  also  participates  in  the  design  phase  of  new 
products  under  development  by  the  vendor. 

In  this  situation,  the  TPM  vendor  becomes,  in  effect,  a  partner 
of  the  manufacturer. 

INPUT  uses  the  term  OEM  TPM  to  describe  this  business. 

•  In  contrast  to  end  user  TPM,  OEM  TPM  is  an  emerging  new  business 
opportunity.  There  are  several  premises  upon  which  the  business  can  be 
approached: 

Offering  services  based  upon  specialized  expertise;  e.g.,  CAD/CAM, 
communications  systems,  Knowledge  of  specialized  process  environ- 
ments, etc. 

Offering  services  based  on  geographic  distribution. 
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Offering  services  based  on  extensive  available  facilities  such  as  system 
support  centers,  repair  depots,  autonnatic  test  equipment,  warehouse 
locations,  transportation  facilities,  communications  networks,  etc. 

•  OEM  TPM  is  receiving  increased  attention  from  major  vendors,  both  as  buyers 
and  sellers. 

Some  large  companies  are  seriously  considering  awarding  OEM  TPM 
contracts  on  some  of  their  smaller  product  lines. 

A  few  of  the  same  companies  are  seriously  considering  entering  the 
OEM  TPM  business  in  certain  geographic  areas. 

•  A  new  phenomenon  that  may  impact  the  TPM  business  over  the  next  few  years 
is  the  interest  evidenced  by  a  few  large  users  in  getting  into  the  business. 

A  large  aerospace  company,  which  maintains  its  own  systems,  is  now 
offering  end  user  TPM  in  a  limited  geographic  area. 

A  large  bank  which  maintains  its  own  office  equipment  is  seriously 
considering  the  creation  of  a  subsidiary  which  would  offer  both  end  user 
and  OEM  TPM  services. 

•  Although  TPM  accounts  for  less  than  10%  of  miaintenance  expenditures  today, 
OEM  TPM  is  growing  rapidly.  As  companies  begin  to  view  services  as  a  profit 
or  loss  center,  especially  within  large  in-house  field  service  organizations, 
INPUT  expects  to  see  more  activity  in  OEM  TPM  in  the  next  few  years. 

•  A  separate  class  of  TPM  vendor  is  one  who  will  take  full  responsibility  for 
maintaining  facilities  of  which  computer  equipment  is  only  a  part. 

These  organizations  are  most  often  found  at  government  installations 
and  at  both  process  and  discrete  manufacturing  plants. 
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Typically,  they  will  maintain  (in  addition  to  electronics)  electrical 
systems,  electromechanical  systems,  sensors,  communications  facilities 
and,  in  a  few  cases,  even  provide  janitorial  services. 

Examples  of  this  type  of  vendor  are  GE,  RCA,  TRW,  Honeywell  and  IBM 
(Federal  Systems  Division). 

Vendor  stability  is  a  key  selection  criteria  for  these  "facilities  managers" 
because,  in  most  cases,  it  is  impossible  to  arrange  for  backup  if  the  vendor 
quits  or  goes  out  of  business. 

TRAINING  TRENDS 

To  maintain  a  cadre  of  service  personnel  capable  of  skillfully  servicing 
munitions  plant  control  systems  over  a  20-year  life  cycle  under  dormant 
conditions  is  a  formidable  challenge. 

The  equipment  is  likely  to  become  obsolete  over  that  timeframe 
causing  manufacturers  to  withdraw  some  or  all  of  their  support. 

Contractor  personnel  come  and  go. 

Learning  retention  (see  Chapter  7,  Section  B)  becomes  extremely 
difficult. 

The  only  solution  to  these  problems  is  to  establish  a  training  program  that  can 
provide  long-term  continuity  regardless  of  personnel  shifts. 

The  retention  of  skilled  personnel  is  also  a  function  of  the  challenge  of 
the  work  environment. 

Every  training  method  carries  with  it  a  difference  in  commitment,  cost,  and 
other  advantages/disadvantages.  This  section  examines  the  following  four 
common  training  methods  in  use  today: 
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Lecture/lab. 

Self-study. 

Video. 

Computer  Based  Instruction  (CBI). 
a.  Lecture/Lab 

i.  Advantages 

•  Lecture  is  the  most  common  method  of  education  in  field  engineering  used 
today.  It  is  also  the  easiest  method  because  of  its  short  development  time  and 
low  resource  requirements. 

The  development  cycle  of  a  lecture/lab  course  is  approximately  eight 
hours  of  writing  for  every  one  hour  of  lecture  time. 

Due  to  the  interaction  between  the  students  and  the  instructor,  greater 
student  interest  is  maintained,  providing,  of  course,  the  instructors  are 
of  high  quality. 

•  Lecture/lab  best  fits  the  basic  education  courses  because  new  employees  can 
be  evaluated  easily.  Due  to  the  low  development  costs,  lecture/lab  is  the  least 
expensive  method  of  training  small  numbers  of  students. 

ii.  Disadvantages 

•  There  are  disadvantages  to  this  method: 

The  quality  of  courses  is  often  inconsistent  because  of  instructor 
capabilities. 
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The  length  of  classes  varies  because  of  extraneous  information  dis- 
cussed. 


Lecture/lab  requires  dedicated  instructors  for  the  duration  of  the 
course. 

Resurrecting  courses  becomes  difficult  because  of  the  reassignment  of 
instructors  -  an  acute  problem  in  a  long-term  program. 

iii.  Costs 

The  cost  of  a  20-day  lecture/ lab  class  today  runs  about  $22,000  exclusive  of 
student  salaries.  The  average  class  size  is  eight  people. 

If  the  program  were  run  continuously,  annual  costs  would  run  about 
$240,000  for  a  one-year  period. 

Although  the  up-front  costs  of  lecture/lab  are  low,  the  ongoing  costs 
become  prohibitive  if  large  groups  must  be  trained. 

b.        Self-Study  Method 

Self-study  is  an  extremely  valuable  training  method,  but  it  can  rarely  stand 
alone.  Self-study  can  be  used  very  effectively  in  situations  where  students  are 
learning  differences  about  products  for  which  they  have  already  been  trained. 
Also,  interspersed  with  other  training  methods,  self-study  increases  the 
effectiveness  of  that  method. 

The  most  widely  used  self-study  method  is  linear,  where  the  information  is 
simply  transposed  into  sections  and  tests.  Many  of  the  other  methods  of  self- 
study  offer  remedial  training.  Their  development  time  can  be  up  to  four  times 
that  of  the  linear  method  because  they  are  very  difficult  to  write. 
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i.  Advantages 


•  Once  the  self-study  course  is  written,  it  is  the  most  inexpensive  field  training 
nnethod.  When  used  with  other  training  methods,  such  as  video  or  self-study,  it 
is  extremely  effective.  Also,  self-study  tends  to  compress  the  course  because 
the  extraneous  information  found  in  the  classroom  is  deleted.  Industry 
standards  estimate  a  2:1  time  compression.  Some  immediate  advantages  of 
self-study  are: 

The  quality  of  courses  is  consistent. 

Large  numbers  of  students  are  handled  quickly. 

Courses  are  available  anywhere  in  the  field  at  any  time. 

ii.  Disadvantages 

•  A  major  disadvantage  of  self-study  is  the  time  required  to  develop  a  self-study 
package.  The  linear  method,  which  simply  takes  information  and  transposes  it 
into  sections,  requires  approximately  26  hours  of  writing  for  each  hour  of 
instruction. 

•  it  is  very  difficult  to  measure  or  control  the  effectiveness  of  self-study.  No 
records  of  who  has  been  trained,  or  their  performance  level,  can  be  made. 

•  Another  difficulty  with  the  self-study  course  is  that  when  it  is  administered  in 
the  field  the  proper  time  or  facilities  required  may  not  be  allocated. 

•  Students  also  experience  difficulty  in  understanding  and  retention  because  of 
the  lack  of  instructor-student  interaction. 

iii.  Costs 


•  Since  each  hour  of  self-study  requires  26  hours  of  writing  (this  is  based  on  the 
linear  method),  a  20-day  course  would  require  520  days  of  development  time, 
or  ^,160  hours.  A  breakdown  of  the  total  price  of  o  20-hour  course  is: 
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4, 1 60  hours/ 1  60  hours 


26  months 


Instructor  costs,  26  months 


$60,658 


Printing  costs,  estimated 


$12,000 


Total 


$72,658 


From  a  straight  cost  standpoint  it  appears  to  be  very  attractive.  However,  the 
following  questions  need  to  be  answered. 

Is  a  lab  required  for  the  course? 

If  there  is  a  serious  maintenance  problem  in  the  field,  how  quickly  can 
it  be  resolved? 

How  are  the  results  of  the  course  going  to  be  validated? 

Who    is   available    if    the   student    has   problems   understanding  the 


The  solutions  to  these  questions  can  increase  the  cost  of  a  self-study  course 
considerably. 


Video  is  one  of  the  fastest  growing  methods  of  training  large  groups  spread 
over  several  locations.  The  recent  advances  in  technology  and  the  ability  to 
interact  with  microprocessors  micke  the  potential  of  video  training  tremen- 
dous. 

In  field  engineering,  a  majority  of  the  information  is  technical,  requiring  that 
the  producers  of  video  courses  be  extremely  knowledgeable  about  the  equip- 


material? 


c. 


Video  Courses 
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ment  taught,  the  functions  involved  in  the  jobs  of  an  FE,  and  the  video  medium 
itself.  This  results  in  a  large  investment  of  time  and  money  when  putting 
together  a  training  staff. 

There  ore  very  few  independent  companies  specializing  in  video  education  for 
field  engineering.  Those  that  do  specialize  in  video  courses  should  be  used  not 
only  to  develop  video  courses  but  to  assist  in  the  development  of  in-house 
video  departments. 

There  are  several  other  forms  of  media  education.  Unfortunately  their  uses 
are  not  as  flexible  as  video.  For  example,  audio-slide  presentation  is  excellent 
for  teaching  large  groups  but  one  is  limited  to  what  can  be  shown  on  a  slide. 

Audio  tapes  are  easy  to  carry  and  can  be  used  in  a  variety  of 
applications.  However,  with  no  visual  capabilities,  students  have  a 
difficult  time  retaining  the  information. 

Due  to  its  growing  popularity  and  potential  impact,  only  video  will  be 
addressed. 

i.  Advantages 

The  advantages  of  video  far  outnumber  the  disadvantages,  as  the  following  list 
indicates. 

Video  matches  the  entertainment  media.  Thus,  students  used  to 
watching  television  are  receptive  to  it. 

Information  is  compressed  4:1,  resulting  in  shorter  classes. 

Expenses  are  reduced  by: 

Less  student  time  at  the  educational  center. 


Eliminating  travel  completely  in  many  cases. 


The  quality  of  courses  is  consistent  and  can  be  updated  very  easily. 

New  microprocessor  systems  allow  video  to  interact  with  the  student. 

0-         Facility  requirements  can  be  stabilized. 

Training  on  discontinued  products  can  be  ongoing,  with  little  or  no 
impact  on  the  field. 

ii.  Disadvantages 

The  major  disadvantage  of  video  is  the  high  front-end  expense  of  equipment 
and  staff  personnel. 

Video  also  requires  a  long  development  cycle  -  about  50  hours  of 
development  for  one  finished  hour  of  video. 

Video,  like  the  self-study  courses,  is  very  difficult  to  manage  in  the 
field. 

iii.  Costs 

The  cost  of  preparing  a  20-day  lecture  course  on  video  tape  today  averages 
approximately  $175,000. 

d.        Computer  Based  Instruction 

Computer  Based  Instruction  (CBI)  has  been  used  very  effectively  by  both  IBM 
and  CDI  (Control  Data  institute)  over  the  past  several  years.  It  not  only  has 
all  the  advantages  of  interactive  self-study,  it  also  handles  many  of  the 
management  functions  which  limit  the  self-study  method. 
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i.  Advantages 

•  The  advantages  of  computer-based  instruction  courses  are: 

Time  spent  at  the  education  center  is  reduced  significantly. 

Large  numbers  of  students  can  be  handled  simultaneously. 

Remedial  education  is  already  built  into  most  systems. 

Education  schedules  can  be  flexible  because  instructors  and  classrooms 
need  not  be  scheduled. 

Software  education  can  be  done  totally  from  any  location  because  the 
remote  terminal  will  simulate  any  environment  needed. 

Most  systems  offer  management  tracking  and  automatic  enrollment. 

Courses  can  be  corrected  and  updated  at  the  central  site  quicl<ly. 

Student  comments  and  questions  can  be  monitored  from  the  central 
site,  allowing  instructor-student  interaction. 

Measurement  results  are  available  at  the  central  site  to  validate  course 
results. 

Tests  and  quizzes  can  be  changed  to  eliminate  cheating. 

ii.  Disadvantages 

•  The  disadvantages  of  CBi  are  as  follows: 

Hardware  system  costs  are  high. 
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A  long  development  cycle  is  required  -  100  hours  of  development  to  one 
hour  of  instruction. 

Students  tire  of  communicating  with  a  terminal. 

Lack  of  visuals  prevents  certain  types  of  information  from  being 
effectively  taught  on  terminals. 

System  terminal  networks  are  required. 

iii.  Costs 

The  cost  of  CBI  is  extremely  difficult  to  calculate. 

if  a  terminal  network  is  in  place,  then  only  course  development  and 
monitoring  are  required.  However,  if  the  purchase  of  a  terminal 
network  is  necessary,  the  cost  will  obviously  be  significantly  higher. 

Companies  who  are  not  equipped  to  put  a  network  in  place  still  have  the 
CBI  option  by  using  outside  learning  centers,  such  as  those  offered  by 
CDI.  These  centers  are  located  throughout  the  country  and  will  also 
assist  in  course  development. 

Regardless  of  resources,  course  development  time  must  be  considered  as  a 
primary  cost  item.  Course  development  calculations  are  based  on  100  hours  of 
writing  for  a  one-hour  course.  This  would  result  in  16,000  hours  of  develop- 
ment, or  1,000  days,  for  a  20-day  course.  So,  16,000  hours  divided  by  160  hours 
per  month  equals  100  months,  or  over  four  man-years.  This  adds  up  to 
development  costs  of  $224,000  at  today's  instructor  salaries. 

At  first  glance,  CBI  seems  to  be  extremely  expensive.  However,  eight 
students  in  a  lecture/lab  course  cost  $14,880  in  expenses  alone.  Therefore,  the 
development  costs  of  CBI  are  more  than  paid  for  after  16  classes. 
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CBI  also  offers  the  potential  for  simulation  of  the  electronnechanical  and  fluid 
(dry  or  wet)  processes  found  in  nnanufacturing  plants,  in  addition  to  its 
inherent  ability  to  duplicate  the  digital  processes  of  central  computers  and 
electronics. 

In  some  cases,  special  purpose  simulators  have  been  built.  They  can  be 
standalone  training  devices  or  units  interfaced  to  a  generalized  CBI 
system. 

With  this  capability,  a  technician  can  be  trained  to  service  an  entire 
system. 

ESCALATION  PROCEDURES 

The  term  escalation  procedure  is  used  to  describe  the  process  by  which  a 
customer  service  problem  "escalates"  through  a  vendor's  command  hierarchy 
until  it  is  resolved. 

The  largest  companies  (IBM,  AT&T,  DEC,  etc.)  usually  have  very  rigid 
and  tightly  controlled  escalation  procedures. 

Other  firms  have  them,  but  they  tend  to  be  loosely  defined  and 
structured,  leaving  a  great  deal  of  decision-making  in  the  hands  of  the 
on-site  serviceperson. 

Every  maintenance  vendor  surveyed  has  a  formal  escalation  procedure  in 
place.  With  epithets  ranging  from  "alert"  procedures  to  "early  warning" 
procedures,  all  have  common  objectives. 

Some  procedures  ensure,  as  tradition  demands,  that  the  account  sales 
representatives  are  alerted  to  down  machines  and  potential  image 
problems  before  they  or  their  managers  are  called  by  irate  customers. 
This  fundamental  objective  dates  back  to  earlier  times  when  field 


service  managers  were  less  confident  in  personally  defusing  potentially 
explosive  situations. 

Some  of  the  procedures  are  oriented  toward  direct  access  to  technical 
assistance.  Field  engineers  are  required  to  discuss  problems  with 
branch  or  regional  specialists  after  a  checkpoint  in  elapsed  time.  The 
intermediate  specialist  is  then  required  to  involve  a  national  or  plant 
technical  support  specialist  after  a  certain  elapsed-time  checkpoint. 

Other  vendors  structure  escalation  procedures  along  management  lines 
following  time  and  circumstance  checkpoints,  at  which  points  increas- 
ing levels  of  management  make  judgements  on  the  delegation  of 
technical,  hierarchical  responsibilities  and  communications  with  the 
user's  hierarchy. 

The  more  advanced  escalation  procedures  involve  checks  and  balances. 

Technical  hierarchy,  field  engineering  management  hierarchy 
and  sales  hierarchy  all  escalate  through  their  own  functional 
lines  of  communication.  Cross  talk  at  each  level  ensures  that 
escalation  occurs. 

Daily  "status  reports"  are  wired  or  called  into  headquarters 
through  the  hierarchy,  with  negative  or  "no  situations"  forced  to 
ensure  that  escalation  channels  are  open  each  workday. 

With  remote  maintenance  aids  such  as  remote  diagnostics,  system  support 
centers,  etc.  becoming  more  prevalent,  strict  adherence  to  formalized  escala- 
tion procedures  becomes  a  necessity  if  the  process  is  to  work  at  all. 

SPARE  PARTS 

With  rising  interest  rates,  computer  vendors  have  become  concerned  over  the 
cost  of  carrying  spares  inventories  which  may  run  as  high  as  40%  of  the  basic 
cost  of  spares  on  an  annual  basis. 
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Third-party  maintenance  vendors  and  autonomous  field  service  divisions 
who  must  expense  their  own  carrying  costs  have  become  particularly 
affected  and  in  many  cases  have  reached  a  "pain  threshold." 

Rapidly  changing  technology  and  the  shorter  product  life  cycles 
(compared  with  earlier  products)  compound  the  problem  by  increasing 
the  risk  of  obsolescence  in  carrying  excess  spares. 

•         The  only  "out"  for  the  vendor  is  to  transfer  the  cost  of  spares  to  the  customer. 
This  is  being  done  with  some  frequency  today  and  is  definitely  a  future  trend. 

In  this  case,  the  customer  pays  up-front  for  spares  and,  in  many 
instances,  stocks  them  on-site. 
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IV       AMMUNITION  PLANT  EPCS  STATUS 


A.  GENERAL  INFORMATION 

•  INPUT  visited  seven  ammunition  plants  that  had  installed,  or  were  planning  to 
install,  digital  computers  for  controlling  some  of  the  production  and/or  process 
lines.  These  seven  plants  are  generally  representative  of  the  types  of 
ammunition  plants  in  the  U.S.  Army  ammunition  production  complex,  as  shown 
in  Exhibit  IV- 1. 

•  The  seven  plants  visited  are  owned  by  the  U.S.  Army  but  are  operated  by 
private  contractors,  (government  owned,  contractor  operated  (GOCO)). 

•  This  section  summarizes  the  information  obtained  from  each  plant.  Specific 
conclusions  and  recommendations  will  be  found  in  other  sections  of  the  report. 

B.  JOLIET  ARMY  AMMUNITION  PLANT 

1.       PLANT  DESCRIPTION 

•  The  Joliet  Army  Ammunition  Plant  (JAAP)  is  devoted  primarily  to  the 
production  of  trinitrotoluene  (TNT)  for  use  by  other  Army  ammunition 
production  facilities. 
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EXHIBIT  lV-1 


U.S.  ARMY  ARMAMENT  MATERIEL 
READINESS  COMMAND 


GOGO 
PLANTS 


ACTIVE 


CRANE 

MC  ALESTER 

HAWTHORNE 


STAND-BY 


ALABAMA** 


HEADQUARTERS 
ARRCOM 


P&E 


ACTIVE 


HOLSTON 
RADFORD 


STAND-BY 


BADGER 

INDIANA* 

JOLIET* 

NEWPORT* 

PHOSPHATE 

DEV  WORKS 

SUNFLOWER 

VOLUNTEER 


DUAL  CAPABILITY 
BEING  EXCESSED 
UNDER  CONSTRUCTION 
(DUAL  CAPABILITY) 

UNDERLINE-  VISITED  BY  INPUT 


COCO 
PLANTS 


SMALL  ARMS 


ACTIVE 


LAKE  CITY 


STAND-BY 


TWIN  CITIES* 


METAL  PARTS 


ACTIVE 


SCRANTON 

RIVERBANK 

MISSISSIPPI*** 


STAND-BY 


GATEWAY 
HAYS 

ST.  LOUIS 
LOUISIANA* 
TWIN  CITIES* 


L/A/P 


ACTIVE 


INDIANA* 
IOWA 
KANSAS 
LONE  STAR 

LONGHORN 

LOUISIANA* 

MILAN 

MISSISSIPPI*** 


STAND-BY 


CORNHUSKER 
NEWPORT* 
JOLIET* 
RAVENNA 
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A  small  portion  of  the  complex  loads,  assembles  and  packs  (LAP)  40mm 
ammunition. 

Some  dinitrotoluene  (DNT)  is  also  produced  at  this  plant. 

•  There  are  16  production  lines  at  Joliet  for  TNT  production. 

Ten  are  batch  production  lines. 

Six  are  CIL  (Canadian  Industries  Ltd.)  continuous  process  lines. 

Three  of  the  six  are  built  to  run  under  an  electronic  process  control 
system  utilizing  direct  digital  control  in  a  closed  loop  mode. 

•  JAAP  is  operated  by  Uniroyal  for  the  U.S.  Army. 

2.       ELECTRONIC  PROCESS  CONTROL  SYSTEM  DESCRIPTION 

•  The  primary  system  designer  for  the  electronically  controlled  portion  of  the 
lines  is  The  Foxboro  Company  of  Foxboro,  Massachusetts. 

The  heart  of  the  system  is  a  Foxboro-designed  PCP-88  system  with 
analog  and  manual  backup. 

A  Foxboro  Spec  200  system,  while  not  part  of  the  production  process, 
does  monitor  the  alarms  and  will  shut  the  system  down  safely  in  an 
emergency. 

•  The  system  was  designed  in  1972  to  1973  and  accepted  by  the  plant  contractor 
and  U.S.  Army  in  1975. 

The  specific  Joliet  design  was  done  in  1972  to  1973,  but  the  PCP-88 
system  for  controlling  continuous  TNT  production  lines  was  designed  in 
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early  1970  for  the  Volunteer  Army  Ammunition  Plant  (VAAP).  The 
Joliet  system  and  Volunteer  system  are  twins. 

At  Joliet  only  one  line  was  tested  even  though  the  system  was  designed 
to  handle  three  lines. 

Closed  loop  control  is  exercised  by  monitoring  flow  speeds,  vat  levels  and 
agitation  speeds.  Valve  positions  are  sensed  and  adjustments  made  if 
necessary. 

There  is  no  on-stream  analysis  for  measuring  chemical  quality. 

The  processors  used  by  Foxboro  in  the  PCP-88  system  are  PDP-8/ls  built  by 
Digital  Equipment  Corporation  (DEC). 

The  other  equipment  in  the  control  system  was  built  by  and/or 
integrated  by  Foxboro  into  the  PCP-88  system.  Exhibit  IV-2  illustrates 
the  principal  components  of  this  EPC  system. 

The  standard  software  for  the  system  was  supplied  by  DEC.  The  FOCUS 
operating  system  is  used  as  a  base. 

Foxboro  supplied  its  standard  EPCS  software  and  also  supplied  custom 
software  where  necessary.  ICI  Americas  wrote  the  EPCS  software 
specification. 

Programs  are  stored  on  the  drum.  Backup,  permanent  storage  for  the 
programs  is  on  mylar-coated  paper  tape. 

Hardware  documentation  is  to  the  board  level  with  some  chip  layouts  detailed. 

This  documentation  is  up-to-date  and  complete  by  all  industry 
standards. 
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The  documentation  was  not  entirely  complete  prior  to  initiation  of  the 
MMT  layaway  program. 

•  Software  documentation  is  complete. 

•  The  operating  environment  and  the  layaway  environment  are  the  same. 

The  temperature  is  kept  at  about  70°F,  and  the  humidity  at  40%  or  less. 

In  summer,  the  air-conditioning  cools  the  system  down  so  completely 
that  occasionally  it  must  be  reheated. 

The  air  is  electrostatically  filtered. 

•  The  computer  room  is  an  earth-protected,  bunker-type  building. 
3.       OPERATING  EXPERIENCE 

•  Line  16,  the  EPC  continuous  process  TNT  production  line,  was  operated  for  six 
months  before  being  shut  down. 

The  maximum  production  rate  of  60  tons  per  day  was  reached  after  two 
days  of  running. 

•  The  startup  and  operating  experience  on  this  line  was  very  good  during  the  six- 
month  period. 

A  temperature  converter  had  a  bad  potentiometer  at  startup  time. 

Some  sensors  froze  during  the  wintertime,  and  they  were  changed  to 
magnetic-flow  meters. 

•  The  system  ran  24  hours  per   day.     No   maintenance  plan   was  actually 
implemented  during  the  operational  period. 
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The  Foxboro-recommended  maintenance  plan  was  available,  but  it  I 
wasn't  used.  ' 

4.       LAYAWAY  AND  REACTIVATION 

•  The  Joliet  Army  Ammunition  Plant  has  been  one  of  the  test  sites  used  by 
ARRADCOM  in  examining  the  effects  of  various  storage  conditions  on  system 
reactivation.  i 

:l 
] 

For  the  three-year  period,  1977-1980,  JAAP  was  reactivated  every  six 

months.  | 

•  During   the   dormant   period   the   electronic   control   system   was  shut   off  ^ 
completely. 

i 

The  proper  environment  was  maintained  in  the  computer  room.  • 

The  magnetic  drum  was  turned  off,  but  the  helium  environment  was 
maintained. 

Those  pneumatic  controls  that  might  have  been  under  water  were 

removed  and  stored  in  a  controlled  environment.  i 

The  24V  relay  logic  room  was  originally  left  alone,  but  the  underground 
level  became  flooded  once  and  the  humid  atmosphere  caused  problems. 
Since  then  a  sump  pump  and  dehumidifier  have  been  used  successfully. 

Some  electromechanical  devices  were  stored  off-line,  and  some  in  the 
24V  relay  building. 

Pumps  left  on-line  were  kept  in  a  preservative. 

•  Software  is  kept  on  the  magnetic  drum  and  on  oil-impregnated  mylar  tape. 
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The  paper  tapes  are  kept  in  the  computer  room.  Another  set  is  kept  at 
JAAP  in  a  vault,  and  a  third  set  is  at  Foxboro. 

The  paper  tapes  at  JAAP  have  been  verified;  the  tapes  at  Foxboro  have 
not  been  tested. 

Documentation  is  kept  in  the  computer  room  in  the  controlled  environment. 

The  hardware  and  software  documentation  also  exist  on  microforms 
which  are  kept  in  the  vaults. 

Reactivation  testing  of  the  system  means  testing  computers,  electronic 
controls,  sensors,  valves,  etc.  without  the  use  of  live  chemicals. 

Sensors  are  tested  randomly  with  a  goal  of  10%  of  the  total  being 
tested. 

Each  control  line  is  tested  separately  and  every  activator. 

All  computer  peripherals  are  tested. 

The  layaway  and  reactivation  testing  program  at  Joliet  has  been  directed  and 
funded  through  ARRADCOM,  who  has  monitored  and  abetted  all  aspects  of  the 
program. 

At  the  last  reactivation  test,  an  ARRADCOM-produced  reactivation 
manual  was  also  tested.  This  is  an  extremely  detailed  manual  aimed  at 
allowing  relatively  untrained  but  skilled  personnel  to  activate  the 
system. 

The  manual  is  an  illustrated  step-by-step  guide  to  turning  the  system 
on.  Each  step  is  described  and  accompanied  by  photographs  showing  the 
cabinet  in  which  the  hardware  necessary  for  the  step  is  located;  the 
specific  switch,  etc.,  is  marked  in  yellow. 
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In  addition,  each  page  contains  a  schematic  of  the  room,  with  locators 
indicating  the  position  of  the  specific  cabinet  within  a  cabinet  bank. 

•  A  series  of  other  user-oriented  manuals  for  check-out,  documentation,  cross 
reference,  indexing,  systems  overview,  and  systems  description  are  underway 
as  part  of  the  ARRADCOM  program. 

•  Overall,  the  reactivation  experience  during  the  three-year  test  period  was 
very  good  with  improvements  made  during  each  cycle. 

At  an  early  reactivation,  a  wire  pit  in  the  24V  relay  building  was 
flooded  and  the  wires  became  grounded.  The  conduits  were  also 
flooded. 

Turning  the  power  on  all  at  once  resulted  in  a  diode  matrix  blowing 
which  is  not  a  recurring  problem. 

All  sensors  are  removed  and  10%  are  tested  and  recalibrated  each  time 
the  system  is  reactivated. 

5.       SPARE  PARTS  MANAGEMENT 

•  There  is  no  spare  parts  management  program  at  JAAP. 

The  spares  on  hand  are  basically  the  Foxboro-recommended  spares. 

They  are,  for  the  most  part,  in  an  environmentally  controlled  area. 

The  spares  are  not  dated  upon  receipt  and  are  not  recycled  when  their 
shelf  life  is  exceeded. 

Spares  are  not  tested  when  received  and  there  have  been  failures  when 
the  spares  have  been  plugged  in. 
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Parts  listing  down  to  the  board  level  have  been  supplied  by  Foxboro. 


The  chips  are  from  many  sources  and  each  is  stamped  with  a  descriptive 
number. 

No  list  is  kept  of  alternative  sources  of  spare  parts  nor  is  there  any 
program  to  obtain  and  test  "compatible"  spares. 

Foxboro  has  been  relied  upon  to  provide  the  necessary  spares,  and  its 
support  has  been  very  good. 

Foxboro  has  announced  that  it  will  not  formally  guarantee  support  of  the 
spares  program  beyond  December  1982. 

PERSONNEL  SKILLS  PROGRAM 

There  is  no  program  to  keep  a  cadre  trained  in  the  maintenance  and  operation 
of  the  EPC  system,  although  there  is  a  personnel  resources  program  sponsored 
by  ARRCOM  which  may  resolve  this  issue. 

Currently  at  JAAP,  only  one  person  is  fully  trained.  He  Is  training  one 
other  person. 

In  a  maintenance  mode  it  is  very  difficult  to  keep  good  people  and  to 
keep  them  motivated  for  extended  periods  of  time.  Fully  trained,  EPC 
people  are  in  demand  and  will  tend  to  find  more  interesting  employ- 
ment. 

At  JAAP,  the  maintenance  force  includes  four  electricians  and  four 
instrument  mechanics.  There  are  no  electronic  technicians  for  the 
system. 

The  documentation  is  in  "ladder  logic"  form  oriented  toward  the 
available  technical  skills. 
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•  Training  in  both  hardware  and  software  has  come  fronn  Foxboro.  Some  DEC 
classes  have  been  attended  for  CPU  training. 

While  some  training  aids  have  been  produced  for  some  of  the  lines, 
there  is  a  minimal  amount  for  the  EPC  lines,  and  training  is  of  the  OJT 
variety. 

Training  will  be  on  an  ad  hoc  basis,  and  difficulty  is  anticipated  in 
attracting  new  people.  Affirmative  action  programs  tend  to  slow  down 
the  finding  and  training  of  new  personnel. 

•  Trained  personnel  can  be  tracked  within  Uniroyal  itself  in  case  of  an 
emergency,  but  there  is  no  tracking  of  personnel  who  leave  the  company. 

There  is  a  difference  of  opinion  as  to  whether  or  not  personnel  who 
have  been  removed  from  the  system  for  six  months  or  more  will  be  able 
to  get  back  "on  board"  in  less  time  than  it  would  take  to  train  fresh 
people.  Respondents  from  other  organizations  feel  that  the  skills  are 
never  completely  lost,  and  the  basic  understanding  of  the  system  is  of 
prime  importance. 

7.       FUTURE  PLANS 

•  The  problems  of  timely  reactivation  are  recognized  by  the  operating  con- 
tractor. 

There  is  concern  over  the  termination  of  vendor  support,  of  maintaining 
and  repairing  an  obsolete  system,  of  the  lack  of  skilled  personnel,  and 
of  the  lack  of  training  materials  with  which  to  train  new  personnel. 

•  The  contractor  has  received  some  funding  to  begin  addressing  the  problems 
that  will  mutually  occur  at  other  AAPs  that  are  using  Foxboro  PCP-88s  when 
Foxboro  withdraws  support  in  December  1982. 
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ARRCOM  has  requested  that  the  three  affected  AAPs  jointly  prepare  a 
recommendation.  ARRADCOM  is  participating  in  this  effort. 

•  Joliet  itself  is  planning  to  follow  with  another  reactivation  in  12  months. 

The  reactivation  testing  program  was  prepared  by  ARRADCOM. 

•  At  the  time  of  the  interview,  the  future  was  an  unknown  because  no 
respondent  knew  how  much  money  would  be  allocated,  if  any,  for  specific 
maintenance  of  the  EPC  line. 

C,       LAKE  CITY  ARMY  AMMUNITION  PLANT 

I.        PLANT  DESCRIPTION 

•  The  Lake  City  Army  Ammunition  Plant  (LCAAP)  is  primarily  a  LAP  plant  that 
receives  material  from  other  plants  for  final  assembly. 

The  plant  was  initially  built  in  the  early  I  9A-0s  and  is  currently  operated 
by  Remington  Arms  Co.,  Inc.  for  the  Arm/. 

•  In  1975-1976  a  building  was  refurbished,  and  automated  production  lines  were 
installed  to  perform  the  four  functions  of: 

Case  manufacture. 

Bullet  manufacture. 

Primer  insertion. 

Load  and  assemble. 
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Connecting  all  these  operations  is  a  sophisticated  materials  handling  system 
that  ends  in  a  packing  and  palletizing  operation. 


The  automated  production  facilities  were  designed  in  1969  but  not  installed  for 
another  five  to  six  years. 

Portions  of  the  installation  are  still  undergoing  acceptance  testing  and 
prove-out. 

Nine  different  vendors  are  involved  in  the  system  design,  fabrication,  installa- 
tion and  equipment  supply  for  all  five  automated  functions,  as  shown  in  Exhibit 
IV-3. 

ELECTRONIC  PROCESS  CONTROL  SYSTEM  DESCRIPTION 

There  are  four  functional  submodules  in  the  manufacturing  of  5.56mm 
ammunition. 

Each  submodule  is  made  up  of  five  production  lines. 

Each  production  line  is  controlled  by  its  own  processor,  as  shown  in 
Exhibit  IV-4. 

a.        Case,  Bullet,  Load  and  Assemble  Submodules 

These  three  submodules  contain  fifteen  production  lines  some  of  which  are 
controlled  by  their  own  DEC  PDP-I  i/05  computers. 

The  basic  production  machinery  shown  in  Exhibits  IV-5  through  iV-7 
consists  of  multiple  rotary  turrets  connected  by  transport  chains  which 
carry  material  from  one  turret  to  the  next.  Each  turret  has  2^ 
machining  stations. 


EXHIBIT  IV-3 


CONTRACTORS  INVOLVED  IN  THE  DEVELOPMENT 
OF  THE  AUTOMATED  PRODUCTION  FACILITY  AT  LCAAP 
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FMC  AUTOMATION 

HONEYWELL 

CASE  MANUFACTURING 
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DATA  GENERAL 

LOAD  AND  ASSEMBLE 
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BATTELLE  NORTHWEST 

DATA  GENERAL 

MATERIAL  HANDLING 

RISI 

SQUARE  D 

MANAGEMENT  INFOR- 
MATION SYSTEM 

SCIENCE  APPLICATIONS 

DEC 
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EXHIBIT  IV-4 


SIMPLIFIED  FACTORY  AUTOMATION  LAYOUT 
AT  LAKE  CITY  ARMY  AMMUNITION  PLANT 


CASE  SUBMODULE 
LINE  5 


CASE  SUBMODULE 
LINE  1* 


CASE  SUBMODULE 
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CASE  SUBMODULE 
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CASE  SUBMODULE  \ 
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•  The  PDP-I  1/05  controls  and  monitors  the  production  and  ejection  of  parts-in, 
ports-out,  and  ejects  by  tool  stations. 

The  information  is  displayed  on  video  terminals. 

•  The  computer  program  models  the  physical  operation  of  the  machine. 

Two  DEC  operating  systems  are  used:  CAP- 1  I  (cassette  programming 
system)  and  RT-I  I  (single  user  on-line  program  development  system). 

Applications  are  programmed  primarily  in  assembly  language. 

b.  Primer  Insert  Submodules 

•  Each  production  line  within  the  submodule  shown  in  Exhibit  lV-8  is  monitored 
and  controlled  by  a  Honeywell  H3I6  computer. 

The  computer  keeps  track  of  all  cases  in  the  machine,  records  and 
stores  data  from  the  inspection  devices,  and  rejects  or  unloads  the 
cases  based  upon  the  inspection  data. 

•  The  DAP  I  6  operating  system  is  used,  and  major  portions  of  the  application  are 
programmed  in  assembly  language. 

c.  Automated  Material  Handling  System 

•  This  system  transports  production  components  and  materials  between  sub- 
modules.  The  system  also  provides  for  a  10-hour  production  accumulation  of 
completed  components  between  submodules.  Finished  goods  are  transported  to 
the  packing  operation  for  boxing,  palletizing,  marking  and  strapping. 

These  transport  operations  are  controlled  by  Square  D  programmable 
logic  controllers  with  a  manual  backup  system  in  case  of  malfunction. 
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Each  controller  consists  of  a  processor,  I/O  racks  and  a  data  terminal. 

The  units  are  programmed  in  BASIC  Language.  For  timing  and 
counting,  programming  is  done  through  the  math/logic  unit. 

d.  Cartridge  Case  Measurement  and  Eject  System 

•  This  system,  consisting  of  a  mechanical  case  handler  and  measuring  instru- 
ments controlled  by  a  Data  General  Nova  210,  inspects  cases  received  from 
the  case  submodule  and  rejects  or  accepts  them  based  upon  the  measurements. 

The  measuring  instruments  are  located  throughout  the  handler  path,  and 
the  computer  keeps  a  running  tally  of  the  data  and  sends  reject  signals 
to  the  handler  for  air  reject  of  unacceptable  cases. 

The  operating  system  is  Data  General's  RDOS.  Applications  are 
programmed  in  assembly  language,  FORTRAN  and  extended  BASIC, 

•  Only  one  of  the  five  computers  is  currently  installed  as  a  prototype.  If  the 
tests  are  positive,  and  as  of  this  writing  they  are,  then  the  full  system  will  be 
installed. 

e.  Management  Information  System 

•  The  management  information  system  (MIS)  is  a  central  data  acquisition,  data 
processing,  control  and  communication  system  that  is  designed  to  coordinate 
data  from  the  production  submodules,  the  inspection  system,  and  the  materials 
handling  system. 

The  system  acquires  production  data,  directs  that  process  samples  be 
taken,  and  determines  from  this  data  if  production  is  under  control. 

Management  information,  inventory  information,  and  production  data 
are  reported  upon  request. 
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Two  DEC  PDP-I  l/34s  serve  as  host  computers,  and  six  PDP-I  I /04s  preprocess 
the  data  received  from  the  other  systems. 

The  peripherals  consist  of  two  disk  drives,  one  line  printer,  one  card 
reader,  one  magnetic  tape  drive,  and  several  CRT  terminals  scattered 
throughout  the  building. 

The  following  DEC  software  is  used: 

RSX-I  IM,  a  multitasking  process  control  system. 

DECNET,  a  communications  operating  system. 

MCP,  a  multiprocessing  control  program. 

This  MIS  system  is  currently  installed  and  going  through  checkout. 

Each  of  the  production  lines  has  an  environmentally  monitored  control  room 
which  contains  the  computer  systems  and  all  other  possible  electronics. 

The  MIS  system  has  its  own  centralized,  environmentally  controlled 
room. 

The  documentation  is  not  kept  up-to-date  by  the  vendors. 

DEC  documentation  is  the  most  up-to-date  (hardware  and  software). 

Any  documentation  gaps  at  LAAP  are  currently  being  addressed  in  the 
MMT  layaway  study. 

OPERATING  EXPERIENCE 

Each  production  line  is  capable  of  producing  1,200  shells  per  minute. 
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This  potential  has  been  achieved,  and  the  line  has  averaged  about  1,000 
shells  per  minute. 


•  At  this  point  only  one  line  in  each  submodule  is  operating.  All  the  systems  are 
undergoing,  or  have  undergone,  prove-out  of  one  phase  or  another. 

•  The  systems  are  running  with  about  20%  downtime  per  month  (August  and 
September  1980)  which  is  an  improvement  over  the  24%  experienced  in  July 
1 980. 

The  problems  on  the  production  lines  have  been  mechanical.  , 

•  Downtime  due  to  computer  or  electronic  malfunction  has  been  very  minimal. 

Specifically,  in  September  1980  the  lines  were  never  down  due  to 
computer  failures  and  were  down  only  eight  times  due  to  electronic 
failures. 

•  This  MIS  system  has  turned  out  very  detailed  reports  summing  all  the  failures, 
mean  time  to  repair  (MTTR),  mean  time  between  failures  (MTBF),  etc.  for 
management  use. 

The  information  is  used  to  monitor  and  improve  the  reliability  of  the 
various  components. 

Minimal  attention  has  been  paid  to  these  reports  by  management 
primarily  because  the  system  is  in  prove-out,  and  it  is  very  difficult  to 
separate  startup  problems  from  design  flaws. 

Management  is  concerned  with  the  low  overall  uptime  experienced  and 
has  taken  steps  to  try  to  improve  the  situation. 

•  The  MTBF  for  the  various  submodules  ranged  from  6  minutes  to  44  minutes 
during  September  1980. 
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Case  submodule 


6  minutes 


Bullet  submodule 


12  minutes 


Primer  insert  submodule 


23  minutes 


Cartridge  case  measurement  eject  system 


27  minutes 


Load  and  assemble  submodule 


44  minutes 


Preventive  maintenance  (PM)  plans  have  been  modified  in  an  attempt  to 
reduce  downtime. 

All  systems  receive  PM  daily. 

Before  starting  operations  in  the  morning,  one  hour  of  PM  takes  place. 
In  the  afternoon  the  systems  undergo  one-half  hour  of  PM. 

The  production  lines  can  operate  with  18  of  the  24  stations  on  each  turret 
operating.  If  more  than  six  stations  on  a  turret  are  down,  then  the  whole  line 
is  brought  down. 


Four  production  lines  are  in  layaway  at  the  present  time.  They  have  never 
been  in  operation. 

The  electronics  are  being  cycled  because  it  is  felt  that  they  may 
deteriorate  if  not  heated  up.  There  is  no  evidence  that  this  is  so;  just  a 
feeling  that  electronic  cycling  will  prevent  surface  corrosion  on  pins, 
sockets,  and  other  metallic  surfaces. 


LAYAWAY  AND  REACTIVATION 
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Each  production  line  is  cycled  twice  each  month  provided  there  are 
enough  parts  so  that  it  can  operate.  At  one  cycling  during  the  month, 
product  is  moved  through  the  line. 

The  reactivation  cycle  is  to  be  drawn  out  to  monthly,  then  quarterly, 
and,  perhaps,  if  the  systems  respond  well,  the  cycle  may  be  stretched  to 
annually. 

Sensors  and  other  line  I/O  devices  are  left  in  place. 

The  computers  that  are  in  an  environmentally  controlled  area  are  kept  under 
the  same  temperature  and  humidity  conditions. 

SPARE  PARTS  MANAGEMENT 

An  inventory  control  group  will  maintain  lists  of  vendors  and  spares. 

The  vendor  list  will  not  go  to  the  component  level  nor  will  parts  lists 
detail  all  the  components. 

Mechanical  spares  are  coated  with  cosmoline  to  prevent  corrosion. 

Electronic  spares  are  kept  in  an  air-conditioned  room  by  accident.  It  is  not 
planned  to  keep  spare  parts  in  a  controlled  environment. 

Lists  of  alternative  sources  of  spares  are  maintained,  and  it  is  planned  to  test 
compatible  spare  parts. 

Spare  parts  have  been  obtained  from  the  other  production  lines  in  order  to 
keep  the  operating  line  functioning. 

The  other  lines  are  left  inoperable  for  a  period  of  time  and  may  not  be 
operable  at  any  given  time. 
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PERSONNEL  SKILLS  PROGRAM 


Turnover  of  personnel  has  been  very  low  in  the  EPC  systenns  area. 

This  is  attributed  to  the  newness  of  the  system  and  the  challenge  of 
getting  it  all  to  operate. 

Mechanical  technicians  are  attracted  fronn  within  the  plant  because 
working  on  and  with  the  automated  production  line  is  exciting. 

A  general  electronics  background  that  can  be  obtained  in  the  military  or  in 
school  is  sufficient  for  a  beginning. 

Remington  Arms  feels  it  takes  about  one  year  to  fully  train  an 
electronics  technician  in  the  EPC  system. 

Vendor  schools  and  OJT  are  the  primary  training  programs  used. 

After  the  line  is  proved-out  and  production  ceases,  Remington  Arms  will  have 
to  face  a  different  type  of  personnel  problem  -  not  one  of  training  but  one  of 
personnel  retention. 

FUTURE  PLANS 

As  of  October,  1980,  funds  have  been  requested  to  test  various  layaway 
procedures  and  cycling. 

It  has  been  proposed  that  moisture-proof  hutments  be  built  around  the 
equipment  and  the  submodules,  that  desiccants  be  placed  in  the  control  room, 
and  that  all  this  could  be  sealed  and  left  shut  for  ten  years. 

Hutments  are  collections  of  huts  or  an  encampment. 
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LCAAP  has  stored  electric  motors  for  ten  years  In  the  normal  standard 
environment  with  no  apparent  harm. 

if  the  hutment  turns  out  to  be  impractical,  then  the  equipment  will 
probably  have  to  be  recycled  during  the  layaway  period. 

•  The  ARRADCOM  program  is  addressing  user-oriented  manuals  in  the  bullet 
and  case  lines.  Further  investigation  is  needed  to  establish  a  final  layaway 
plan. 

8.  MISCELLANEOUS 

•  As  long  as  Remington  Arms  has  a  production  contract  for  5.56mm  ammunition, 
each  of  the  five  lines  can  be  activated  cyclically  to  produce  live  products. 

This,  of  course,  means  the  idle  lines  shouldn't  be  used  to  provide  spares 
for  the  active  line. 

D.       LONE  STAR  ARMY  AMMUNITION  PLANT 

I.        PLANT  DESCRIPTION 

•  The  Lone  Star  Army  Ammunition  Plant  (LSAAP),  operated  by  Day  &  Zimmer- 
man, Inc.,  is  a  LAP  COCO  plant  that  can  produce  monthly  on  a  single  shift: 

3,800,000  -  Artillery  primers 

15,000,000  -  Detonators  and  lead  assemblies 

854,700  -  Mortar  and  artillery  fuses 

630,000  -  Grenades 


-  95- 


JNPUT 


63,000  -        '       M72  light  antitank  weapons 

325,000  -  Artillery  shells 

136,000  -  Mortar  shells 

800,000  -  Supplennentary  charges 

The  plant  was  constructed  during  1941-1942  with  14  production  lines. 

During  Vietnam  the  plant  produced  127  different  items  using  12,000 
people. 

LSAAP  is  a  currently  active  plant  with  a  workforce  of  about  2,000  and 
minimal  production  contracts. 

Many  of  the  production  lines  have  undergone  some  modernization,  and  most 
lines  use  programmable  logic  controllers  (PLCs)  for  some  level  of  automatic 
local  control. 

Two  lines  are  in  the  process  of  being  computerized. 

The  E  line  for  the  LAP  functions  on  105mm  ammunition. 

The  P  line  for  the  production  of  detonators. 
E  line  design  began  in  1974,  installation  started  in  1976  and  is  still  continuing. 
P  line  design  began  in  1975  with  installation  starting  in  1976. 
PLC  installations  began  as  early  as  1972. 


-  96  - 


ELECTRONIC  PROCESS  CONTROL  SYSTEM  DESCRIPTION 


The  computerized  E  line  receives  shells  from  other  plants,  processes  powder, 
loads  the  powder  into  the  shell,  then  loads  the  detonator  and  assembles  the 
other  parts  to  make  a  complete  projectile. 

MRC,  Inc.  designed  the  turnkey  system  and  is  currently  doing  the 
installing. 

MRC  selected  a  Taylor  Instruments  control  system  which  is  based  upon 
a  Varian  computer  system  (division  of  Sperry  Univac). 

Most  of  the  PLCs  are  made  by  Allen-Bradley. 

The  E  line  control  system  shown  in  Exhibit  IV-9  uses  a  dual  computer  system 
consisting  of: 

Two  CPUs. 

Two  14"  disk  drives. 

Two  diagnostic  consoles. 

Two  CRT  displays. 

Two  teletypewriter  (TTY)  printers. 

One  magnetic  tape  drive. 

One  line  printer. 

The  computer  monitors  and  controls  about  600  I/O  lines. 
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EXHIBIT  IV-9 


SIMPLIFIED  E-LINE  COMPUTER  CONTROL  SYSTEM  AT 
LONE  STAR  ARMY  AMMUNITION  PLANT 
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There  is  an  extensive  materials  handling  system  to  move  the  parts  from  one 
production  area  to  the  next. 

The  measuring  devices  at  the  end  of  the  I/O  lines  include: 

Pressure  transducers. 

Infrared  temperature  sensors. 

Differential  pressure  gauges. 

Pneumatic  transducers. 

E  line  software  is  primarily  FORTRAN  (MRC  and  Taylor)  with  some  Varian 
assembly  language  used  for  the  basic  Taylor  programs. 

MRC  also  uses  its  own  program  design  language  called  A5TRAN. 

E  line  software  is  the  total  responsibility  of  MRC  and  is  considered  adequate. 

Disk,  cards  and  magnetic  tape  are  used  for  program  storage.  The  card 
decks  cannot  be  updated  because  there  is  no  card  punch  in  the 
configuration.  Program  updates  must  come  from  MRC. 

All  documentation  for  the  E  line  is  through  MRC,  and  the  hardware  documen- 
tation is  checked  to  the  board  level. 

The  manufacture  of  detonators  on  the  P  line  is  controlled  by  a  Modcomp  II 
computer  system  designed  and  installed  as  a  turnkey  package  by  Computer 
Sciences  Corporation. 

The  configuration  consists  of  a  single  CPU,  a  disk  drive,  a  TTY,  and  two 
CRT  displays. 
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There  are  several  PLCs  on  the  line  also. 

There  are  about  120  I/O  points  on  the  P  line  connected  to  either  the  computer 
or  the  PLCs. 

The  sensors  connected  to  the  computer  are  thermocouples  and  hi-lo 
limit  switches. 

Load  cells  are  generally  connected  to  the  PLCs. 
The  Foxboro  Company  Is  the  primary  instrumentation  supplier  on  the  P  line. 

Foxboro  has  provided  documentation  to  the  component  level. 

Software  is  stored  on  the  disk  drive. 

Software  for  the  P  line  has  had  micny  problems,  especially  Interfacing 
with  the  PLCs.  In  general,  this  software  is  currently  considered  less 
than  adequate  (November  1980). 

It  Is  planned  to  keep  both  computer  systems  In  controlled  environments; 
however,  that  hasn't  oeen  fully  achieved  yet. 

The  E  line  computer  room  is  earth-banked  and  has  a  controlled 
environment. 

The  P  line  computer  system  environment  Is  being  upgraded  to  meet 
"computer  system  standards."  Currently,  the  humidity  Is  controlled  to 
some  extent  and  the  air  is  filtered  through  the  cabinets. 

An  environmentally  controlled  system  separate  from  the  building's 
system  is  necessary. 
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•  There  has  been  no  operating  experience  with  either  line  since  both  are  still 
under  construction. 

The  maximum  production  potential  of  the  E  line  is  one  million  projec- 
tiles per  month  working  three  shifts  and  seven  days  per  week. 

•  Currently  there  is  no  preventive  maintenance  plan. 

Maintenance  will  be  done  on  an  "as  needed"  basis. 
4.       LAYAWAY  AND  REACTIVATION 

•  An  extensive  layaway  definitions  and  procedures  manual  is  being  prepared. 
Since  it  was  incomplete  and  unapproved  when  scanned  in  November  1980,  no 
comments  can  be  made  concerning  its  viability. 

The  plan  is  based  upon  a  detailed  definition  of  the  possible  states  of  the 
system  and  the  procedures  to  move  from  one  state  to  another  within  a 
set  of  rules  (time  allotted,  etc.). 

•  Exhibit  IV- 10  shows  the  three  states  -  operating,  preservation  (dormancy), 
confirmation  (testing)  -  and  the  time  allotted  for  changing  the  state  of  the 
system. 

•  Within  this  broad  framework,  a  hierarchical  documentation  structure  can  be 
built  that  could  extend  down  to  the  smallest  component. 

•  Since  the  production  system  is  under  construction  at  present,  it  is  in  a  form  of 
layaway. 

The  materials  handling  system  was  idle  and  becoming  rusty  in  many 
places. 
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EXHIBIT  lV-10 


PLANNED  LAYAWAY  AND  REACTIVATION 
CYCLES  AT  LSAAP 
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Other  idle  equipment  appeared  to  be  left  in  place  with  no  effort  made 
to  inhibit  corrosion. 

•  It  is  felt  that  the  computer  systems  should  be  left  running  in  order  to  prevent 
deterioration. 

•  No  reactivation  has  taken  place  since  no  activation  has  taken  place. 
5.       SPARE  PARTS  MANAGEMENT 

•  There  are  detailed  parts  listings  for  both  the  E&P  lines. 

•  Vendor  listings  are  not  complete  nor  are  alternative  sources  of  parts. 

Spares  are  purchased  from  the  manufacturer  directly. 

•  Spares  for  the  PLCs  are  in  good  supply,  but  there  are  few  computer  parts  in 
stock. 


After  the  systems  are  operational,  spares  will  be  purchased  in  accord- 
ance with  both  the  vendors'  recommendations  and  fiscal  regulations. 

•         Currently  there  is  no  testing  program  for  spare  parts  and  no  dating  program 
for  incoming  spares. 

Day  &  Zimmerman  would  like  to  do  its  own  maintenance  on  the  system, 
down  to  repairing  boards  if  necessary. 


6.        PERSONNEL  SKILLS  PROGRAM 

•  For  the  E  line.  Day  &  Zimmerman  has  trained  people  generally,  but  the 
installation  has  taken  so  long  that  attrition  has  given  the  contractor  some 
concern. 
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Day  &  Zimmerman  is  looking  toward  establishing  its  own  internal 
training  activity. 

For  the  P  line,  since  it  has  been  idle,  there  are  no  trained  D&Z  personnel  in 
either  hardware  or  software. 

An  in-house  training  function  would  aid  this  situation. 

D&Z  likes  to  recruit  people  from  technical  schools  with  four-year  degrees. 

D&Z  tries  to  recruit  systems  people  not  design  engineers. 

Right  now  D&Z  feels  it  has  fancy  equipment  that  no  one  knows  how  to 
use. 

FUTURE  PLANS 

The  prime  objective  is  to  finish  both  lines,  have  them  prove-out,  and  then 
become  productive. 

Another  important  objective  is  to  complete  the  hardware  and  software 
documentation  for  both  lines. 

The  documentation  must  be  accurate,  up-to-date,  and  complete. 

A  plan  must  be  implemented  for  keeping  the  documentation  up-to-date. 

These  are  the  goals  of  the  ARRADCOM  program. 

The  layaway  document  must  be  completed,  approved,  funded  and  implemented. 

It  is  an  inclusive  document  that  deals  with  all  the  electronics  associated 
with  the  modernized  lines  including: 
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Closed  circuit  TV  systems. 
Fire  protection  systems. 

Materials  handling  equipment  control  systems. 
Power  systems. 

Process  machines  control  systems. 
Security  systems. 
Telephone  systems. 

•  ARRADCOM  is  planning  implementation  efforts  similar  to  those  being  done  at 
VAAP  toward  solving  this  problem. 

8.  MISCELLANEOUS 

•  The  layaway  plan  seemed  to  be  complete  in  that  it  wasn't  limited  to  just  the 
EPC  system  but  included  all  other  electronic  systems  in  the  modernized  plant. 

•  A  scanning  of  the  plan  indicated  that  it  was  very  detailed  for  equipment  and 
will  approach  software  in  a  similar  fash  ion,  but  seemed  not  to  be  concerned 
with  the  personnel  problems  during  layaway. 


E.       LOUISIANA  ARMY  AMMUNITION  PLANT 


I.        PLANT  DESCRIPTION 

•         The  Louisiana  Army  Ammunition  Plant  (LAAP)  is  a  metal  parts  machining 
plant.  The  parts  are  sent  to  other  plants  for  LAP  operations. 
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The  overall  plant  was  built  in  1939. 

Installation  of  the  automated  machining  lines  began  in  1976  and  is  still 
going  on. 

There  are  two  automated  lines  in  the  plant.  The  Y  line  produces  105mm 
casings,  and  the  M-4  line  produces  the  ogives. 

Neither  of  these  lines  has  any  centralized  control  system. 

Both  lines  are  automated  using  PLCs  that  were  made  by  14  different 
vendors. 

LAAP  is  operated  by  the  Thiokol  Corporation  for  the  U.S.  Army. 

ELECTRONIC  PROCESS  CONTROL  SYSTEM  DESCRIPTION 

The  Y  line  is  automated  with  14  different  electronic  PLCs  made  by  10 
different  manufacturers,  as  shown  in  Exhibit  IV- 1  1. 

The  A-B  7360s  and  the  A-B  1774  are  complex  PLCs  and  are  very  close 
to  being  computers. 

Even  though  this  is  a  highly  automated  line,  people  are  needed  to  help  at 
virtually  all  workstations. 

The  entire  line,  since  it  is  still  being  installed,  has  never  been  run,  but  part  of 
each  individual  operation  has  been  tested. 

The  primary  sensors  connected  to  all  the  PLCs  are  limit  switches. 

There  are  some  contact  relays  in  the  various  systems  especially  for 
controlling  the  material  handling  subsystem. 
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EXHIBIT  IV-n 


LAAP  Y-LINE  ELECTRONIC  CONTROL  SYSTEMS 


•  CE  LOGITROL  -  FORGE 

•  MODICON  284  -  BAND  WELDERS 

•  ALLEN-BRADLEY  1774  PLC  -  RISI  MATERIALS 

HANDLING  SYSTEM 

•  ALLEN-BRADLEY  1772  PLC-2  -  J&L  FINISH  TURNING 

•  ALLEN-BRADLEY  7360  -  J&L  OGIVE 

•  ALLEN-BRADLEY  7360  W/PAL  OPT lON-OLLOFSON  BASE 

MACHINES 

•  TEXAS  INSTRUMENTS  511  SERIES  -  HARDNESS  TESTER 

•  MODICON  484  -  MAGNAFLUX  TESTERS 

•  RELIANCE  AUTOMATE  31  -  THREADING  MACHINE  CONTROL 

•  AEROTECH  SMART  1  -  THREADING  TOOLING  CONTROL 

•  AMF  VERSATRAN  MODEL  600  -  ROBOTS  ON  SLOTTERS  AND 

ON  JSL  OGIVE 

•  MARPOS  #2000  CONTROL  -  AUTOMATIC  GAUGE 

•  MARPOS  #4000  CONTROL  -  AUTOMATIC  GAUGE 

•  SQUARE  D  LOGIC  CONTROLLOR  -  M-107  NOSE  PRESS 
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•  Hardware  documentation  is  available  to  the  board  level,  but  personnel  on 
board  currently  can  troubleshoot  down  to  the  component  level. 

•  There  is  very  little  "standard"  software  for  the  PLCs.    Each  PLC  is  custom 
programmed  by  the  vendor  for  its  particular  function. 

•  Programs  are  retained  in  the  PLC  itself. 

With  certain  PLCs  the  programs  can  be  stored  on  magnetic  tape 
cassettes. 

Paper  tape  can  be  used  for  program  storage  with  a  particular  PLC  if 
necessary. 

Those  PLCs  that  have  UV  erasable  EPROMS  have  proven  tedious  to 
reprogram.  The  industry  is  moving  away  from  such  PROMS. 

•  The  PLCs  are  designed  to  operate  in  an  uncontrolled  factory  environment. 

t 

The  temperature  in  the  plant  ranges  from  below  20^F  in  winter  to  up  to 
about  I40°F  in  the  summer. 

The  humidity  is  uncontrolled,  as  is  noise,  vibration,  particulate  matter, 
etc. 

•  The  same  temperature  and  humidity  conditions  will  exist  when  the  system  is  in 
a  dormant  state. 

Most  of  the  system  can  be  considered  in  dormant  storage  now  since  it  is 
not  operating  and  is  in  the  seme  position  as  it  would  be  under  layaway. 

3.       OPERATING  EXPERIENCE 

•  The  production  line  hasn't  been  operated  yet. 
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•         There  are  no  plans  for  methodically  recording  all  failures  and  using  the  data 
for  systems  improvement  or  spares  management. 


•  The  overall  maintenance  plan  is  to  follow  strictly  the  advice  of  each  individual 
vendor. 

•  However,  the  ARRADCOM  MMT  plan  is  to  develop  a  Defect  and  Failure 
Reporting  procedure  to  be  used  by  all  EPC  facilities. 

4.       LAYAWAY  AND  REACTIVATION 

•  There  is  no  reactivation  history  with  this  system  yet,  since  it  has  never  been 
operational. 

•  However,  certain  operations  have  been  producing  for  a  while. 

When  they  are  shut  down  for  two  to  four  months,  the  PLCs  are  cycled 
once  each  week. 

•  In  general,  the  PLCs  would  be  laid  away  in  place  or  according  to  the 
manufacturer's  advice.  The  ARRADCOM  MMT  plan  implementation  would 
positively  impact  this  trend. 

There  is  the  possibility  of  removing  the  circuit  boards  from  the  PLCs 
and  storing  them  for  long  periods  in  a  controlled  environment,  but  no 
definite  plans  along  these  lines  exist.  There  is  some  concern  that  unless 
this  were  done  in  a  well  planned  manner,  mixups  would  occur. 

•  The  materials  handling  PLCs  would  be  stored  in  a  controlled  environment. 

•  Moisture  problems  ore  anticipated  with  the  limit  switches,  but  there  are  no 
plans  to  avoid  the  potential  problems. 

This  was  reported  to  INPUT. 
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Cassette  tapes  with  programs  are  stored  in  a  normal  office  environment. 

Some  PLCs  are  remotely  loaded  and  have  self-diagnostics  built  in. 

LAAP  is  trving  to  justify  the  buying  of  automatic  loaders  for  the  PLCs. 

All  documentation  is  on  plain  paper  and  is  kept  in  a  normal  office  environ- 
ment. 

The  ARRADCOM  plan  will  Impact  these  areas  requiring  the  problems  to  be 
addressed. 

SPARE  PARTS  MAi^JAGEMENT 

Spares  are  stocked  according  to  the  vendor's  suggested  list. 
There  is  no  detailed  listing  of  the  parts  vendors. 

There  is  no  list  of  alternative  sources  for  parts. 

There  is  no  testing  of  spares  because  there  is  no  equipment  for  testing 
them. 

Vendor  support  has  been  good.  Turnaround  time  on  board  repairs  is  also 
satisfactory. 

These  repairs  were  generally  done  by  local  distributors  of  the  vendor. 

Other  sources  of  board  repair  have  been  located  and  successfully  used. 

Spare  parts  are  usually  left  in  the  original  wrapping  and  stored  in  the  factory 
environment. 
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6.       PERSONNEL  SKILLS  PROGRAM 


•  After  the  initial  training  which  is  to  be  provided  by  the  vendor,  the  contractor 
is  planning  to  train  all  new  people. 

Vendors  have  delivered  training  vans  to  the  site  for  initially  training 
contractor  personnel. 

An  outside  training  company  has  been  used  for  training  on  the  more 
complex  PLCs. 

•  The  systems  can  be  operated  and  maintained  by  an  industrial  plant  electrician 
or  a  person  trained  in  microprocessors. 

With  the  assistance  of  the  contractor,  two  local  trade  schools  offer 
courses  in  industrial  instrumentation  and  some  students  are  expected  to 
become  employees. 

•  Reactivation  should  be  fairly  easy  for  a  technician  with  some  electronic 
background. 

If  something  goes  wrong,  more  experienced  people  must  be  available. 

•  Experienced  people  from  LAAP,  who  may  have  moved  on  to  other  jobs  with 
the  contractor,  can  generally  be  traced  if  necessary. 

•  Currently,  there  is  no  problem  attracting  trainable  people  from  the  schools 
and  through  employment  ads. 

The  South  is  a  growing  area,  and  as  new  industry  moves  in,  competition 
will  drive  pay  scales  up.  The  contractor  is  concerned  about  his  ability 
to  meet  the  competition. 
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7.        FUTURE  PLANS 


•  The  prime  concern  is  getting  the  line  running,  and  all  resources  are  being 
devoted  to  that  goal. 

Sonne  thought  has  been  given  to  layaway  concerns,  but  no  plan  for  the 
electronic  equipment  is  available  other  than  to  leave  it  in  place. 

The  contractor  would  like  to  have  funds  to  obtain  a  complete  set  of 
spares,  a  good  dating  and  numbering  system  for  the  spares,  and  a 
controlled  environment  for  storing  them. 

8.  MISCELLANEOUS 

•  There  is  some  thought  that  if  the  prime  goal  is  to  maximize  production  in  the 
minimum  amount  of  time,  then  the  production  line  should  be  simplified. 

The  line  should  be  broken  into  thousands  of  very  simple  jobs  so  that 
thousands  of  people  can  be  quickly  trained  to  perform  the  functions 
safely. 

F.       MISSISSIPPI  ARMY  AMMUNITION  PLANT 

1.        PLANT  DESCRIPTION 

•  The  Mississippi  Army  Ammunition  Plant,  operated  by  Mason-Chamberlin  for 
the  U.S.  Army,  is  currently  under  construction. 

Mason-Chamberlin  offices  are  in  a  shopping  mall  where  the  construc- 
tion is  supervised. 

•  The  plant  will  be  a  complete  manufacturing  facility. 
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Facilities  for  the  machining  of  metal  parts  and  for  LAP  operations  will  .  ! 

exist  at  the  site. 

'■'I 

•  Several  vendors  of  electronic  control  equipment  have  already  been  selected,  i 

.1 

and  other  proposals  are  being  evaluated.  Computers  and  PLCs  will  be  used  in  | 
the  plant. 

A  separate  Westinghouse  computer  system  will  control  each  of  two 

forge  systems.  ■ 

Allen-Bradley  controllers  will  run  the  turning  centers.  J 

The  LAP  functions  will  be  computer-controlled  and  are  out  for  bids.  ■! 

•  The  plant  will  make  155mm  ammunition.  | 

j 

The  forge  area  is  the  first  portion  to  be  built.  This  area  will  do  shell  ^ 
forging  and  machining. 

2.       ELECTRONIC  PROCESS  CONTROL  SYSTEM  DESCRIPTION 

•  A  65K  memory  Westinghouse  W2500  computer  system  will  control  the  forge 
operation.  ,  ~  j 

An  H-P  terminal  with  cassettes  is  the  console.  | 

'i 

Conrac  monitors  will  be  used  for  information  display.  i 

A  line  printer  will  be  on-line.  ' 

I 

Allen-Bradley  and  Modicon  PLCs  will  be  used  to  control  and  operate  the 
machining  centers. 
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A  closed  circuit  TV  system  along  with  video  recorders  will  be  used  all  over  the 
plant  to  monitor  production  line  operations. 

The  Westinghouse  system  will  monitor  about  1,200  I/O  points. 

Proximity  switches  and  photoelectric  relays  will  be  used  in  addition  to 
weight  and  temperature  sensors. 

Little  detailed  hardware  documentation  is  currently  available. 

The  documentation  is  stored  with  the  computer  and  is  general,  rather 
than  detailed  board  layouts,  etc. 

Westinghouse  is  supplying  all  software. 

The  operating  system  is  the  Westinghouse  W2500  standard  operating 
system. 

Westinghouse  is  also  supplying  the  necessary  custom  software  written  in 
FORTRAN. 

All  software  will  be  kept  on  cassettes. 

Five  copies  of  the  software  currently  exist. 

The  operating  environment  for  the  computer  will  be  controlled  and  will  meet 
the  vendors'  specifications. 

There  will  be  a  generator  to  power  the  computer  room  separately  from 
the  rest  of  the  plant. 

The  dormant  environment  is  planned  to  be  controlled  also,  but  the  system  will 
have  to  have  the  power  turned  off. 
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•  Currently,  the  system  is  in  a  very  dornnant  state.  Since  nnoney  was  allocated 
for  the  system  and  other  equipment  at  the  beginning  of  construction,  the 
system  has  been  sitting  in  a  warehouse  for  over  a  year. 

The  portion  of  the  warehouse  where  the  computer  and  other  electronic 
equipment  is  stored  has  controlled  temperature  and  humidity. 
However,  only  an  unanchored  plastic  curtain  separates  the  environ- 
mentally uncontrolled  portion  of  the  warehouse  from  the  controlled 
portion. 

3.  OPERATING  EXPERIENCE 

•  There  was  no  operating  experience  as  of  the  Fall  of  1980,  since  the  plant  is 
still  under  construction. 

•  Mason-Chamber  I  in  plans  to  maintain  the  system  itself  using  the  Westinghouse 
preventive  maintenance  plan  as  a  base. 

The  plan  is  to  have  four  or  five  technicians  in  the  computer  area  while 
the  plant  is  operational. 

Diagnostics  can  be  run  on  each  !C  board  and  for  each  I/O  line  on  the 
board. 

4.  LAYAWAY  AND  REACTIVATION 

•  The  Westinghouse  system  is  currently  stored  in  its  original  packing  containers 
in  a  partially  environmentally  controlled  warehouse.  It  is  in  effect  "laid 
away." 

•  After  the  plant  has  "proven  out"  the  plan  is  to  lay  the  computer  away  with 
power  off  but  to  maintain  the  proper  computer  room  environment. 
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Even  though  there  has  been  no  reactivation  experience  with  this  system,  the 
following  two  itenns  seem  important: 


Westinghouse  recommends  that  to  60  days  be  planned  to  power-up 
the  system  especially  the  power  supplies  and  capacitors  -  two  critical 
components.  If  the  contractor  has  only  90  days  to  bring  the  plant  up  to 
full  production,  waiting  one-half  to  two-thirds  of  the  allotted  time  just 
to  get  a  computer  started  leaves  the  contractor  very  little  time  if 
something  goes  wrong. 

The  computer  system  is  currently  in  a  dormant  state.  When  bringing 
the  system  up  for  the  first  time,  many  hardware  problems  will  occur. 
An  extensive  failure  monitoring  system  should  be  funded  to  try  to  glean 
as  much  information  from  the  experience  as  possible.  Attempts  should 
be  made  to  try  to  separate  the  failures  due  to  storage  and  storage 
conditions  from  those  due  to  first-time  use. 

•  Electromechanical  devices  will  be  left  in  place,  covered  and  protected. 

•  The  magnetic  cassettes  and  other  magnetic  storage  media  will  need  periodic 
recycling  because  the  media  will  deteriorate  with  time. 

•  Software  will  be  stored  during  dormancy  in  vaults  in  the  computer  area  and 
main  manufacturing  area. 

5.       SPARE  PARTS  MANAGEMENT 

•  No  formal  program  has  been  established  since  the  plant  is  under  construction. 

A  program  as  yet  undefined  will  be  organized  when  the  electronics  shop 
is  set  up. 

•  The  contractor  has  a  detailed  listing  from  Westinghouse  of  every  type  of 
component  in  the  system. 
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The  contractor  has  alternative  sources  for  spares  because  all  com- 
ponents are  standard  off-the-shelf  items. 

Currently,  there  is  no  program  to  test  spares  as  they  are  received. 

6.  PERSONNEL  SKILLS  PROGRAM 

•  Job  specifications  are  currently  being  drawn  up  for  all  operating  personnel. 

There  is  a  local  NASA  installation,  and  the  contractor  expects  to  be 
able  to  attract  trained  personnel. 

•  The  contractor  is  planning  to  use  standard  classroom  training  and  OJT  for  all 
new  personnel. 

7.  FUTURE  PLANS 

•  Ail  programs  are  actually  future  plans  for  this  plant. 

•  The  contractor  is  planning  to  videotape  the  running  of  all  equipment  and  use 
this  in  training  programs  and,  more  importantly,  for  reactivation  purposes. 

•  It  is  planned  to  run  the  diagnostics  bimonthly  during  plant  operation,  and  at 
the  same  cycle  clean  the  contacts  on  the  IC  boards  of  the  PLCs. 

8.  MISCELLANEOUS 

•  The  contractor  is  aware  of  the  requirement  to  draw  up  a  layaway  plan  for  the 
entire  plant  and  to  pay  particular  attention  to  plans  for  the  electronic 
equipment. 

As  of  November  1980,  little  had  been  done  along  those  lines,  since  the 
primary  objective  of  all  personnel  is  to  get  the  plant  built  and 
operating. 
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•  The  ARRADCOM  program  is  also  expected  to  impact  this  issue  of  layaway. 
G.       RADFORD  ARMY  AMMUNITION  PLANT 


I.        PLANT  DESCRIPTION 

•  The  Radford  Army  Ammunition  Plant  (RAAP)  is  operated  by  Hercules,  Inc. 

The  plant  was  built  about  1942  as  an  explosive  powder  and  TNT 
manufacturing  facility  and  has  been  operating  continuously  ever  since. 

•  Two  automated  production  lines  are  being  built  -  one  is  in  prove-out  and  the 
other  is  under  design  and  construction. 

The  CASBL  (Continuous  Automated  Single  Base  Line)  system  was 
designed  in  1965,  the  installation  was  completed  in  1975,  and  it  is  still 
in  prove-out.  It  had  not  been  started  up  as  of  November  I  980. 

The  CAMBL  (Continuous  Automated  Multi-Base  Line)  is  currently  under 
construction. 

Both  lines  will  produce  propellants. 

The  single-base  line  (CASBL)  is  designed  to  produce  2,500,000  pounds  of 
single  base  propellent  per  month  in  a  continuous  process.  The  multi- 
base  line  (CAMBL)  combines  several  chemicals  in  the  process  of  making 
propellent. 

•  The  CASBL  process  line  was  designed  by  the  Computer  Sciences  Corporation 
(CSC)  and  modified  by  Hercules,  Inc. 
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ELECTRONIC  PROCESS  CONTROL  SYSTEM  DESCRIPTION 


The  CASBL  process  control  system  uses  a  Foxboro  PCP-88  computer  as  a  base 
to  control  the  successive  operations  of  propellent  mixing,  extruding,  cutting, 
drying  and  packing  for  shipment,  as  shown  in  Exhibit  IV- 1 2. 

The  computer  system  uses  a  pair  of  DEC  PDP-8/ls  as  the  CPUs,  disk 
drives  for  mass  storage,  and  a  variety  of  peripherals,  as  shown  in 
Exhibit  IV-13. 

The  PCP-88  system  is  different  than  the  VAAP  and  JAAP  systems, 
primarily  in  the  I/O  complement.  This  system  is  card-  and  disk- 
oriented  while  the  others  are  paper  tape-  and  drum-oriented. 

In  this  system  there  are  about  90  direct  digital  control  loops.  The  system 
monitors  320  analog  measurements  and  provides  digital  status  and  control  for 
over  700  points. 

Data  from  the  sensors  is  sampled  periodically,  with  the  period  deter- 
mined by  the  sensor  function. 

Typically  in  this  system,  the  sample  interval  is  one  second  for  a 
flow  loop,  five  seconds  for  a  pressure  or  level  loop,  and  30 
seconds  for  a  temperature  loop. 

The  supervisory  computer,  which  is  the  backup  computer,  interrupts  the  DDC 
processor  once  per  second  to  read  the  process  variable  from  the  core  memory 
of  the  DDC.  These  variables  are  stored  on  the  supervisory  disk  providing  a 
very  warm  backup  system  in  case  the  DDC  has  a  failure. 

Additional  system  backup  is  provided  by  an  extensive  analog  system 
which,  in  turn,  is  backed  up  by  a  manual  system. 


EXHIBIT  IV-12 


PROCESS  SAMPLING  POINTS  -  CASBL  MANUFACTURING  AREA 


ALCOHOL 


5%  M  C 
SLURRY 


VACUUM 

BENT  FILTER 


GRAVIMETRIC 
WEIGH 


SCREW  EXTRUDER 


WATER.VEY  SOAVATER 

DRY  system! 


DDC  LOOPS  -  10 
ANALOG 
MONITOR 

DIGITAL 
STATUS 

IN 

OUT 


14 


64 

56 


DDC  LOOPS  -14 
ANALOG 
MONITOR  -18 

DIGITAL 

STATUS 

IN  -  40 
OUT   -  30 


DDC  LOOPS  -21 

ANALOG 
MONITOR 

DIGITAL 
STATUS 
IN 

OUT 


•41 


-42 
-  42 


DDC  LOOPS  -16 

ANALOG 

MONITOR 

DIGITAL 
STATUS 

IN       -  26 

OUT  -76 


DDC  LOOPS  -18 
ANALOG 
MONITOR  -ISI 

DIGITAL 

STATUS 

IN  -  305 
OUT  -  336 


WATER  VEY 
SYSTEM 


-  120  - 


INPUT 

YMDCR 


n 
< 


1° 


(/I 

-i  1/1  -it 
<  t  <  s. 


1/1 

o  u 
I-  o 


LLI 

> 

t/) 

q: 

ro 

LU 

1 

1- 

> 

D 

Q. 

m 

U 

I 

X 

< 

LU 

D 

G 

-J 

CD 

(/) 

< 

PHIC 
LAY 

(/)  LU 

<  =- 

 »" 

0  Q 

UJ 

-  121  - 


INPUT 

YMDCR 


Groups  of  failure  detection  routines  are  run  every  second  as  part  of  the 
continuous  control  software  package.  They  are  capable  of  detecting 
any  out-of-limit  conditions  within  the  set  point  control,  valve  control, 
data  entry,  and  analog  conversion  modules.  A  failed  module  will  be 
disabled,  the  operator  alerted,  and  If  necessary  shift  to  one  of  the 
backup  systems. 

There  is  some  feeling  that  the  supervisory  system  is  not  a  full  backup,  if  there 
is  an  I/O  problem  or  a  failure  in  the  data  base,  bus  buffer,  or  interface,  both 
systems  go  down. 

The  systems  are  not  as  completely  separable  as  they  should  be  in  a  true 
backup  system  design. 

in  addition  to  the  PCP-88  equipment,  the  automated  lines  include: 

An  Allen-Bradley  PLC  which  is  used  to  more  precisely  control  the 
operational  and  safety  aspects  of  the  plant  motor  and  solenoid  valve 
control  and  to  assume  some  of  the  more  iterative  plant  "housekeeping" 
tasks. 

A  CCTV  system  which  is  used  to  monitor  the  entire  production  line. 

A  telephone  system  which  provides  voice  communications  along  the 
process  line. 

Standard  software  was  supplied  by  Foxboro  with  the  operating  system  origi- 
nally furnished  to  Foxboro  by  DEC. 

CSC  wrote  the  applications  programs  and  Hercules  rewrote  most  of 
them. 

The  programs  are  written  in  Foxboro's  language  BATCH.  Some  reports 
are  written  in  FORTRAN,  and  the  data  acquisition  program  is  in  BASIC. 
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Programs  are  primarily  stored  on  disk.  Backup  program  storage  is  on  verified 
paper  tape. 

Paper  tapes  are  stored  in  the  control  room  and  eventually  a  copy  may 
be  kept  in  a  vault.  The  paper  tape  is  of  standard  quality. 

Hardware  documentation  is  down  to  the  board  level.  System  documentation  is 
extensive. 

Software  documentation  consists  primarily  of  assembly  language 
listings  and  BATCH  listings. 

The  CAMBL  system  is  similar  to  a  bakery  manufacturing  process. 

Four  Foxboro  Fox  I/A  computers  run  four  complex  manufacturing  towers. 

Each  computer  complex  has: 

One  line  printer. 

Two  CRT  displays. 

One  card  reader. 

One  disk  drive. 

Each  system  has  about  1,000  I/O  lines  to  sensors  and/or  actuators,  etc. 
The  sensors  measure  pressure,  temperature,  and  moisture  content. 

Standard  and  custom  software  is  being  written  by  Foxboro. 

All  software  at  this  point  is  kept  on  removable  disks. 

Documentation  is  not  extensive  at  this  time  (November  1980). 
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System  documentation  is  the  operations  manual. 

Logic  prints  to  the  board  level  are  not  defined. 

Standard  software  has  a  fully  annotated  assembly  listing. 

Custom  software  documentation  is  a  function  of  the  language  used,  and 
source  listings  are  the  extent  of  the  documentation. 

The  computers  for  both  lines  are  in  a  standard  environmentally  controlled 
room. 

On  the  CASBL  system,  the  interface  and  motor  control  rooms  have 
strict  temperature  controls  and  heat  vents. 

On  the  CAMBL  system,  there  is  danger  from  acetone  and  alcohol,  so  all 
loops  are  "safe"  loops  with  explosion-proof  contacts. 

OPERATING  EXPERIENCE 

There  has  been  no  operating  experience  with  either  production  line  yet. 

CASBL  is  in  prove-out. 

CAMBL  is  under  construction. 
The  potential  propellent  production  from  the  lines  is: 

CASBL  -  1.25  million  pounds  per  month. 

CAMBL  -  2.4  million  pounds  per  month. 

As  of  November  1980,  one-half  of  the  CASBL  system  had  been  checked  out. 
The  second  half  has  yet  to  be  calibrated  or  checked  out. 
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The  CASBL  system  has  a  minimum  record  of  system  failures. 

There  is  a  log  book  of  failures  and  a  log  book  of  major  changes,  but  no 
one  is  really  in  charge  of  a  "system"  or  overall  "plan"  for  incorporating 
the  changes  into  all  the  documentation. 

No  one  exercises  the  responsibility  of  tracking  failures  In  such  a  way 
that  a  change  in  preventive  maintenance  procedures  might  pick  up  the 
potential  for  failure  before  it  occurs. 

Computer  memory  has  been  the  most  troublesome  area  at  this  CASBL 
installation. 

In  the  CASBL  system,  the  preventive  maintenance  plan  calls  for  monthly 
maintenance  by  the  plant  maintenance  department. 

This  plan  was  implemented,  but  it  provides  less  maintenance  than  the 
vendor  recommended. 

Due  to  increased  problems,  the  preventive  maintenance  plan  has 
reverted  back  to  the  vendor's  recommendation  for  specific  equipment, 
notably  the  paper  tape  reader. 

No  real  thought  has  been  given  to  maintenance  plans  or  procedures  on  the 
CAMBL  system. 

There  is  some  feeling  that  Foxboro  may  be  needed  to  repair  the  system. 
LAYAWAY  AND  REACTIVATION 

There  has  been  no  reactivation  experience  at  either  installation,  nor  hove  any 
systems  been  put  in  storage.  Systems  did  not  operate  for  some  period  during 
installation,  but  that  is  not  considered  layaway. 
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There  are  several  ideas  concerning  how  the  CASBL  system  should  be 
stored,  but  no  definitive  plan  has  been  accepted  and  approved. 


It  was  felt  that  layaway  and  reactivation  should  be  handled  by  the  mainte- 
nance crew. 

Most  of  this  could  be  handled  by  the  maintenance  department  if  the 
procedures  were  written  with  this  goal  in  mind. 

Most  I/O  gear  should  be  left  in  place,  and  those  needing  recalibratlon 
could  probably  be  tested  in  place. 

The  line  should  be  reactivated  starting  at  the  manual  level,  proceed  to 
the  analog  system,  and  then  to  the  digital  system.  This  helps  isolate 
problems  to  the  most  recently  activated  portion. 

The  ARRADCOM  program,  as  the  VAAP  plan,  may  be  dovetailed  into  the 
CASBL  and  CAMBL  plans. 

SPARE  PARTS  MANAGEMENT 

For  the  CASBL  system  a  listing  of  the  major  parts  exists.  The  plan  was  to  go 
into  more  detailed  listings  of  both  components  and  vendors,  but  the  man  in 
charge  left  and  as  of  November  1980  no  one  had  taken  on  that  job. 

Spare  parts  for  CASBL  are  stored  in  the  manufacturer's  packaging. 

Spares  are  not  dated  or  tested. 

Board  testers  are  not  available,  so  parts  have  to  be  tested  and  faults 
diagnosed  while  the  system  is  running. 

The  equipment  is  old,  and  there  are  generally  few  spare  boards. 


-  126  - 


INF 


The  Allen-Bradley  PLC  has  a  complete  set  of  spares. 

Vendors  will  not  supply  generic  parts  information,  so  it  is  very  difficult  to  find 
alternative  sources  of  parts. 

Some  feel  that  Hercules  will  have  to  design  its  own  boards  after 
Foxboro  drops  support  for  the  PCP-88. 

The  CAMBL  system  is  still  under  warranty.  It  might  be  out  of  warranty  before 
it  can  be  installed. 

Extended  warranty  is  available,  but  the  cost  is  extremely  high. 

Hercules  has  a  list  of  spare  parts  suggested  by  Foxboro  and  the  Corps  of 
Engineers. 

Hercules  would  like  to  get  generic  part  descriptions  from  Foxboro. 
Foxboro  has  relinquished  some  proprietary  drawings. 

#  , 

Spares  for  the  Fox  I/A  system  are  readily  available  since  the  system  is 
still  being  marketed. 

Hercules'  purchasing  unit  gets  parts  and  instruments  when  necessary  from 
whatever  source  can  supply  the  right  item. 

Spares  would  be  recycled  if  shelf  life  were  critical,  but  no  spares  have 
been  so  designated  yet. 

There  is  no  planned  testing  of  spares  for  the  CAMBL  system. 
PERSONNEL  SKILLS  PROGRAM 

The  CASBL  system  will  probably  have  a  trained  and  dedicated  technical  crew 
when  it  goes  into  production. 
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There  have  been  training  sessions  for  technicians  but,  in  general,  there 
have  been  few  ways  for  them  to  nnaintain  their  skills. 

The  CAMBL  technical  people  go  through  extensive  training  by  Foxboro  for  18 
weeks,  eight  hours  per  day  for  five  days  per  week. 

Beyond  the  above  training,  there  is  no  plan  for  attracting  and  keeping  skilled 
technicians  to  learn,  run,  and  maintain  the  system. 

During  the  prove-out  and  initial  operations  phases  of  the  system  there 
is  sufficient  challenge  to  attract  skilled  people.  The  problems  may 
occur  after  the  systems  have  been  reliably  producing  for  a  few  years 
and  the  initial  people  leave  or  are  transferred. 

Hercules  has  been  working  with  the  local  community  college  to  set  up 
electronic  courses  that  will  be  pertinent  to  the  needs  at  RAAP. 

After  these  courses,  it  still  takes  at  least  six  months  in  classes  and  OJT 
to  train  a  good  CASBL  operator. 

There  are  plans  to  use  various  simulation  training  techniques  in  both  CASBL 
and  CAMBL  training  courses. 

Personnel  from  within  the  plant  have  been  trained  and  plans  are  to 
rotate  them  through  all  programs. 

This  produces  highly  salable  people  and  the  attrition  rate  may  increase. 
FUTURE  PLANS 

The  future  of  layaway  and  reactivation  is  cloudy  at  this  point.  The  immediate 
future  that  must  be  addressed  is  to  get  CASBL  productive  and  finish  the 
construction  of  CAMBL  so  it  can  go  to  prove-out. 
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The  lack  of  trained  personnel  is  a  long-term  problem  and  limited  steps 
are  being  taken.  ' 

8.  MISCELLANEOUS 

•  In  looking  toward  layaway  and  reactivation,  one  problem  may  be  CASBL 
documentation  which  should  be  re-evaluated  and  brought  up  to  date. 

•  The  CAMBL  system  may  experience  some  of  the  obsolescence  problems  that 
the  PCP-88  system  is  having. 

The  Fox  1/A  system,  while  now  being  sold  as  part  of  the  new  Spectrum- 
compatible  series,  is  an  older  system  that  was  incorporated  into  the 
Spectrum  line. 

It  is  not  compatible  within  the  spectrum  line  as  are  some  of  the  newer 
processors. 

The  interfacing  of  the  I/O  lines  and  peripheral  equipment  is,  however, 
much  more  compatible  within  the  Spectrum  family  than  is  possible 
within  the  PCP-88  line,  which  is  a  dead  end. 

As  of  this  writing  it  is  understood  that  the  Fox  I/A  system  will  be 
replaced  with  a  "new"  system. 

H.  VOLUNTEER  ARMY  AMMUNITION  PLANT 

I.  PLANT  DESCRIPTION 

•  The  Volunteer  Army  Ammunition  Plant  (VAAP)  is  an  8,000-acre  TNT  manufac- 
turing facility  operated  by  ICI-Americas  for  the  U.S.  Army. 
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The  plant  was  initially  built  in  the  1941-19^2  period  by  the  Corps  of  Engineers. 

The  EPC  system  was  designed  in  1971  and  accepted  in  1974. 
The  plant  has  10  TNT  production  lines. 

Four  are  batch  production  lines. 

Six  are  CIL  continuous  process  lines  with  four  of  these  lines  under 
direct  digital  control. 

The  entire  plant  is  laid  away  and  is  in  standby  status. 

Only  one  DDC  line  was  ever  operated,  and  it  produced  TNT  for  about  15 
months  before  being  shut  down. 

ELECTRONIC  PROCESS  CONTROL  SYSTEM  DESCRIPTION 

The  EPC  system  at  Volunteer  is  a  PCP-88  system  from  The  Foxboro  Company 
and  was  designed  jointly  by  Foxboro  and  ICI-Americas  (formerly  Atlas 
Chemical  Industries). 

The  system  is  a  twin  of  the  one  at  JAAP;  a  simplified  computer 
diagram  was  shown  in  Exhibit  IV-2. 

A  Foxboro  Spec-200  system  is  used  for  backup  alarm  monitoring  and  emer- 
gency line  shutdown. 

The  system  includes  both  analog  and  manual  backup  equipment. 

The  sensors  and  instruments  are  electronic  except  for  the  valve  actuators 
which  are  pneumatic. 

1/0  equipment,  other  than  the  control  lines,  is  minimal. 


Magnetic  drums  are  used  for  mass  storage. 

Paper  tape  readers  and  a  paper  tape  punch,  along  with  typewriter-type 
printers,  make  up  the  rest  of  the  peripheral  equipment  complement. 

The  software  is  DEC  and  Foxboro  standard  programs  plus  custom  work  by 
Foxboro,  expanding  the  capability  of  the  system  to  handle  10  lines. 

The  scan  rate  was  changed  from  one  second  to  two  seconds  permitting 
more  lines  to  be  scanned  in  the  interim. 

Machine  language  and  FORTRAN  II  were  used  for  the  programs. 

The  programs  are  stored  on  the  drum  and  on  paper  tape. 

Hardware  documentation  exists  in  great  detail. 

It  was  found  in  training  classes,  however,  that  the  drawings  did  not 
match  the  actual  physical  installation. 

Software  documentation  is  felt  to  be  more  accurate  than  the  hardware  and 
just  as  complete. 

The  operating  environment  is  computer  "standard"  -  temperature  kept  in  the 
low  seventies  and  humidity  in  the  low  forties. 

The  system  seems  to  be  temperature  sensitive  and  performs  unpredict- 
ably if  the  temperature  goes  over  80*^F. 

The  air  is  double  filtered  to  ensure  protection  against  acid  fumes 
getting  into  the  computer  room  as  happened  on  one  occasion. 
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OPERATING  EXPERIENCE 

During  the  15  months  the  line  was  operating,  production  averaged  50  tons  per 
day.  .  ^ 

The  maximum  capability  of  the  line  is  70  tons  per  day  which  was 
achieved  several  times. 

The  entire  system  averaged  80%  uptime  while  the  computer  averaged  98% 
uptime. 

There  were  some  initial  problems  that  couldn't  be  specifically  isolated, 
but  adding  wiring  shields  eliminated  the  problems  and  the  system  ran 
smoothly. 

A  daily  log  is  kept  which  tracks  all  interaction  with  the  system  including 
problems  and  fixes. 

An  overall  preventive  maintenance  program  was  in  effect  while  the  system 
was  operating. 

There  appeared  to  be  no  need  to  modify  the  program  since  the  system 
ran  smoothly. 

Preventive  maintenance  scheduling  is  now  computerized. 

Some  changes  were  made  to  improve  system  reliability  during  the  production 
running  period. 

• 

Some  sensors  failed  and  had  to  be  redesigned.  Specifically,  the 
differential  pressure  cells  wore  out  the  orifices  and  magnetic  flow 
meters  were  substituted  for  the  cells. 
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A  mixed  acid  analyzer  and  gas  chromatograph  were  added  on-line,  but 
the  line  was  shut  down  before  they  could  be  tested. 


LAYAWAY  AND  REACTIVATION 

The  DDC-TNT  line  was  shut  down  in  March  1977,  and  the  computer  was  turned 
off  in  July  1979  based  upon  guidelines  of  the  ARRADCOM  layaway  plan. 

The  computer  environment  was  kept  the  same  as  it  was  when  in  operation. 

The  relay  racks  and  the  Spec  200  are  also  kept  in  a  controlled 
environment. 

Some  sensors  and  other  remote  I/O  devices  were  removed  and  placed  in 
a  controlled  environment. 

Some  electronic  devices  were  kept  in  place  and  desiccants  were  kept  in 
the  cabinets. 

Most  valves  were  left  in  place.  Those  in  the  outside  settling  areas  were 
put  in  uncontrolled  storage. 

The  helium  environment  of  the  magnetic  drum  was  maintained. 

All  documentation  is  kept  in  the  computer  room  and  in  a  vault. 

The  system  was  reactivated  under  the  ARRADCOM  layaway  plan  after  nine 
months  in  March-April  1980. 

Several  small  problems  occurred  that  in  combination  turned  out  to  be 
major  problems. 

The  problems  were  mainly  with  some  iC  chips  especially  with  the  pins 
and  sockets. 


33  - 

INPU 


Some  of  these  problems  might  have  been  generated  by  the  nitric  fumes 
accident  a  few  years  earlier. 

Also,  in  the  nine-month  period  people  became  unfamiliar  with  the 
details  of  the  system,  and  this  prolonged  the  reactivation  time. 

The  PCP-88  system  was  reactivated  again  in  September  1980  for  a  training 
class  sponsored  by  the  ARRCOM  personnel  resources  program.  This  reactiva- 
tion was  much  less  traumatic  than  the  first. 

SPARE  PARTS  MANAGEMENT 

There  is  no  dedicated  program  aimed  at  compiling  and  maintaining  a  list  of  all 
vendors  whose  components  are  used  in  the  system. 

An  incomplete  ad  hoc  list  exists. 

A  formal  spare  parts  program  existed  during  production  but  has  lapsed  during 
dormancy. 

Vendor  support  in  this  area  is  good,  but  Foxboro's  inventory  has  become 
depleted  and  now  other  sources  of  spares  must  be  located. 

There  is  no  current  formal  program  to  identify  alternative  sources  of 
components. 

There  is  no  program  to  test  "compatible"  components  or  to  test  spares 
to  see  if  they  are  operational. 

Foxboro  suggested  that  I  CI  or  the  Army  buy  old  PCP-88  systems  as  a  source  of 
spares  as  they  are  replaced  at  other  installations. 
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6.  PERSONNEL  SKILLS  PROGRAM 

•  TNT-making  skills  are  rapidly  vanishing  in  the  U.S.  because  no  TNT  is 
currently  being  made,  nor  do  there  seenn  to  be  plans  to  make  any  soon. 

•  DDC  classes  are  held  regularly  for  new  people  brought  on  board  through 
various  programs. 

Volunteer  does  its  own  training  using  audio/visual  materials  made  by 
ICI-Americas,  and  also  some  supplied  by  Foxboro. 

•  Holding  on  to  people  is  a  major  problem. 

No  steady  sources  of  new  personnel  are  available. 

•  The  Army  has  a  personnel  resources  program  which  keeps  track  of  military 
personnel  by  skill. 

ICI-Americas  tracks  its  own  people  as  long  as  they  remain  with  the 
company. 

•  Foxboro  Is  the  only  training  vendor  for  this  system. 

7.  FUTURE  PLANS 

•  Currently,  the  three  ammunition  plants  that  have  PCP-88  systems  are  jointly 
defining  a  program  outlining  their  alternatives  and  making  recommendations  in 
light  of  the  withdrawal  of  support  for  the  system  by  Foxboro. 

•  An  Army  ammunition  plant  personnel  resources  program  for  the  civilian 
employees  with  attempts  to  track  them  when  they  leave  has  been  talked 
about,  but  sufficient  interest  hasn't  yet  been  aroused  to  fund  such  a  program. 
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The  making  of  televison  tapes  of  the  training  progrann  would  preserve  the 
courses  in  the  best  possible  way  if  professionally  done  and  if  the  PCP-88s 
remain. 

MISCELLANEOUS 

Mention  was  made  of  the  possibility  of  using  the  local  Army  reserve  unit  as  a 
trained  basic  cadre  for  the  system. 

They  could,  after  training,  bring  the  system  up  each  year  as  part  of 
their  active  duty. 


I 
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V   MAINTENANCE  METHODOLOGIES 


V        MAINTENANCE  METHODOLOGIES 


A.  INTRODUCTION 

•  This  chapter  pulls  together  the  information  gathered  from  the  vendor  and  user 
interviews. 

There  is  no  "right"  approach  presented  here,  but  merely  cn  analysis  and 
summation  of  the  views  of  many  respondents  -  sometimes  several  in 
each  of  the  companies. 

•  All  the  respondents  were  cooperative;  some  users  were  confused. 

They  spend  their  time  trying  to  keep  the  system  running  99.9%  of  the 
time,  yet  the  interview  topic  concerned  shutting  the  system  down.  The 
thought  was  very  alien  to  their  thinking. 

Most  vendors  also  hadn't  given  the  layaway  problem  any  thought,  but 
when  the  requirement  was  laid  out  before  them  they  became  intrigued. 
This,  of  course,  doesn't  make  their  responses  any  more  or  less  valid. 

•  Each  type  of  respondent  took  a  different  tack  toward  maintenance.  Some 
responses  were  more  appropriate  than  others,  but  none  could  present  hard  data 
to  support  their  views  which  usually  called  for  a  "play-it-safe"  policy. 
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Only  one  company,  a  leasing  connpany,  could  say  that  systems  can  be  |^ 
stored  under  normal  warehouse  conditions  with  minimal  ill  effects. 


B.       VENDOR  ANALYSIS 

I.       SYSTEM  VENDORS 

•  The  types  of  vendors  interviewed  are  shown  in  Exhibit  V-l.  They  ranged  from 
manufacturers  specializing  in  programmable  controllers  and  industrial  controls 
specifically,  to  manufacturers  of  total  electronic  process  control  systems. 

a.        Status  of  Current  Systems 

•  Current  systems  have  provided  excellent  performance  relative  to  application, 
availability  and  failure  rate.  Some  have  been  operational  until  a  decision  has 
been  made  to  upgrade  the  system. 

•  Design  considerations  varied  with  the  vendor's  marketplace  and  application. 
The  common  goal  of  all  vendors,  however,  was  to  improve  hardware  and 
software. 

Redundancy  was  built  into  areas  of  critical  circuit  functions  in  order  to 
maintain  integrity  of  the  process. 

Certain  critical  functions  were  decoupled,  or  unbundled  using  modular 
design  to  minimize  risks  of  a  common  failure  element. 

The  modular  design  concept  of  the  overall  architecture  was  improved. 
This  concept  places  a  circuit,  or  multiple  circuits  and  their  functions, 
on  individual  plug-in  type  subassemblies.  These  subassemblies  can  then 
readily  be  removed  and  replaced  with  a  known  working  replacement. 
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EXHIBIT  V-1 


TYPES  OF  VENDORS  INTERVIEWED 


VENDOR 

INDUSTRY 

1 

IKAI-hlv^  l-UUINlhKb,  IN UUbT RIAL  SCALES , 
TRUCK/RAIL  WEIGHING  STATIONS 

2 

INSTRUMENTS,  SENSORS,  ACTUATORS 

3 

PROGRAMMABLE  LOGIC  CONTROLLERS 

4 

MINICOMPUTERS  AND  MICROPROCESSORS 

5 

EPC  SYSTEMS 

6 

EPC  SYSTEMS 

7 

EPC  SYSTEMS 

8 

MICROPROCESSOR  MEASUREMENT  AND 
CONTROL  SYSTEMS 

9 

EPC  SYSTEM 

10 

EPC  SYSTEMS 

n 

PROGRAMMABLE  LOGIC  CONTROLLERS 

12 

EPC  SYSTEMS 

13 

EPC  SYSTEMS 

14 

PROGRAMMABLE  LOGIC  CONTROLLERS 

15 

PROGRAMMABLE  LOGIC  CONTROLLERS 
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Current  systems  also  have  self-diagnostic  capability  which  simplifies 
troubleshooting  to  the  defective  module. 

Software  was  simplified  for  the  nontechnical  operator.  This  software  is 
menu  driven,  and  constantly  prompts  the  operator  to  answer  questions 
before  advancing  to  the  next  step. 

Combining  such  functions  as  analog  and  sequential  latch  control  elimi- 
nated the  need  for  multiple  microprocessors. 

Emphasis  on  ladder  logic  has  been  to  simplify  maintenance  for  the 
customer  facility  electrician. 

•  The  decreasing  cost  of  hardware  offers  much  more  product  for  the  money 
compared  to  ten  years  ago.  As  a  result,  tradeoffs  are  less  likely  to  be 
considered. 

The  decreasing  cost  of  hardware  enables  a  vendor  to  be  more  selective 
with  discrete  components  -  improving  equipment  reliability. 

The  modular  design  of  equipment  architecture  does  not  force  a 
customer  to  start  with  a  system  any  larger  than  his  immediate  need,  as 
might  have  been  done  ten  /ears  ago. 

A  customer  with  a  small  need  can  begin  with  a  small  dollar 
investment  and  expand  as  the  need  requires. 

•  Vendors  were  hesitant  to  discuss  MTBF  (Mean  Time  Between  Failures),  since  it 
has  a  different  meaning  to  each  vendor. 

MTBF  formulae  vary  with  each  vendor. 

Design  engineering  is  normally  required  to  estimate  MTBF  for  each 
product  designed.    This  is  intended  to  aid  projections  for  the  critical 
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support  areas  such  as  logistics  and  staffing.  Engineering's  estimates  of 
MTBF  are  frequently  conservative. 

One  vendor's  ennpirical  data  reflected  an  MTBF  that  was  6  to  10 
times  greater  than  expected  by  mil-spec  2I7C. 

Quality  control  traditionally  records  all  historical  data  provided  by 
customer  service.  This  information  is  then  compared  with  the  estab- 
lished MTBF  and  MTSF  formulae  (Mean  Time  to  System  Failure). 

This  equation  is  usually  a  function  of  failure  rate,  repair  rate,  test 
interval,  operate  time,  and  system  redundancy. 

Selected  MTBF  and  MTSF  data,  particularly  low  figures,  are  then  given 
to  engineering  for  possible  design  changes  that  will  improve  the  MTBF 
and  MTSF. 

•         Of  the  15  vendors  interviewed,  only  one  was  mil-spec  oriented. 

INPUT  discovered  that  most  vendors  believed  in  a  willingness  on  the 
part  of  the  user  to  trade  off  higher  equipment  costs  for  improved 
reliability. 

Some  vendors,  however,  could  not  justify  the  increased  cost  associated 
with  mil-spec  on  the  theory  that  a  device  is  only  as  good  as  its  weakest 
link,  and  not  all  components  used  meet  mil-specs. 

In  addition,  vendors  do  not  use  mil-spec  components  because 
they  could  not  then  meet  the  price  competition  in  the  market- 
place. 

Mil-spec  217B's  guidelines  are  used  by  some  vendors  for  MTBF  calcula- 
tions. 
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Packaging  techniques  for  components  varied  with  the  vendor's  targeted 
marketplace,  application  and  environment. 

One  vendor  applies  a  special  varnish  to  PC  boards  to  minimize  the 
effects  of  moisture.  This  procedure,  however,  makes  troubleshooting 
and  repair  more  difficult,  since  the  varnish  is  hard  to  penetrate  with 
test  equipment  probes,  or  the  required  unsoldering  and  soldering  of 
discrete  components. 

Another  vendor  offers  an  optional  cabinet  containing  a  water-heater 
exchanger  for  use  in  locations  with  extreme  environmental  problems. 

The  highly  competitive  components  market  encourages  vendors  to  use 
standard  components  as  a  means  of  maximizing  component  resources. 

Equipment  performance  has  shown  steady  improvement  due  to  the  improved 
development  of  such  components  as  ICs. 

With  IC  functional  component  densities  approaching  50,000  and  100,000 
transistors,  concern  is  being  voiced  over  the  quality  and  performance  of 
the  chips. 

Vendors  had  little  comment  regarding  system  or  subassembly  weaknesses. 
Most  felt  there  were  no  weaknesses  of  significance. 

The  reliability  of  IC  sockets  is  always  of  some  concern  when  ICs  are 
plugged  in,  as  opposed  to  being  soldered  in  place. 

When  ICs  are  plugged  into  a  socket,  there  is  concern  for  two  basic 
variables: 

The  human  element  responsible  for  inserting  the  IC. 
Proper  handling  during  packing,  shipment  and  unpacking. 
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Another  area  of  concern  is  the  proper  design  of  and  manufacturing 
process  for  isolating  sensitive  circuits  from  60-cycle  power  sources. 

Since  most  equipment  operates  at  very  low  voltages,  60-cycle 
noise  can  play  havoc. 

Proper  electrical  grounding  throughout  a  system  is  equally  important  in 
order  to  eliminate  or  minimize  the  effects  of  static  electricity.  If  not 
properly  grounded,  static  electricity  can  prove  hazardous  to  operators 
and  is  also  capable  of  repeatedly  destroying  software  programs  held 
within  memory. 

b.        Support  Plan 

Dramatic  technological  advances  compel  the  vendor  to  constantly  review  its 
maintenance  strategy  in  order  to  ensure  that  it  complements  the  latest 
product.  This  is  particularly  true  of  vendor  service  organizations  that  are 
profit  centers. 

* 

Support  problems  are  greater  for  customers  who  find  it  necessary  to  contract 
overall  system  design  with  multiple  vendors,  because  of  the  difficulty  of 
coordinating  the  combined  efforts  of  the  vendors.  In  such  a  situation, 
individual  vendors  are  insensitive  to  the  customer  need  for  network  coordina- 
tion. 

Given  these  circumstances,  it  is  not  uncommon  for  a  customer  to 
identify  one  vendor  as  the  primary  contractor  assuming  the  responsi- 
bility for  overall  coordination  and  project  management. 

In  all  cases,  the  customer  is  responsible  for  site  preparation  which  includes 
satisfying  environmental  specifications,  power  requirements,  cable  laying  and 
spares  for  the  hardware. 
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Vendors  are  committed  to  some  form  of  support,  primarily  parts  during  the 
warranty  period. 

Vendors  of  the  more  sophisticated  total  electronic  process  control  systems  will 
assume  installation  and  startup  responsibility. 

Vendors  of  the  less  sophisticated  systems,  consisting  of  programmable  logic 
controllers  only,  will  provide  telephone  support  during  installation  and  startup. 
On-site  support  is  provided  on  an  as  required  basis. 

The  less  sophisticated  systems  such  as  standalone  programmable  logic 
controllers  usually  enable  the  customer  to  purchase  parts  locally  from 
vendor  support  locations  or  vendor-appointed  distributors. 

These  programmable  controllers  are  usually  purchased  from  local 
distributors,  who  provide  the  first  line  of  support  including  maintenance 
and  spare  parts. 

Generally,  only  vendors  of  total  EPC  systems  offer  maintenance  contracts  on 
hardware  and  software.  There  are  two  basic  reasons  for  this: 

The  vendor  is  in  the  service  business  and  retains  a  staff  of  qualified 
support  personnel  trained  on  the  product  line. 

The  customer  is  not  in  the  service  business  due  to  the  magnitude  and 
sophistication  of  such  systems,  and  generally  cannot  justify  the  costs 
related  to  establishing  such  a  service  unit. 

All  vendors  make  hardware  and  software  maintenance  training  available. 
Initial  training  usually  occurs  before  delivery,  during  factory  checkout  and 
acceptance,  or  at  times  during  the  warranty  period. 
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The  vendor,  who  obligates  the  customer  to  installation,  startup  and  on- 
going maintenance,  will  provide  timely  training  as  part  of  the  overall 
purchase. 

Vendors  of  the  more  sophisticated  total  systems  will  provide  continuing 
hardware  and  software  training  at  an  additional  cost. 

The  vendor  assumes  that  customer  personnel  sent  for  training  have  a 
basic  understanding  of  electricity  and/or  electronics  for  hardware 
maintenance.  it  is  also  assumed  that  individuals  sent  for  software 
training  have  a  basic  understanding  of  software. 

No  vendor  is  prepared  to  train  neophytes  under  any  circumstances. 

Licensed  distributors  of  vendors  are  also  prepared  to  provide  limited 
training  for  hardware  and  software  maintenance. 

•  Preventive  Maintenance  (PM)  is  usually  identified  with  electromechanical 
devices  such  as  printers,  card  reader/sorter,  and  tape  drives.  Totally 
electronic  devices  such  as  video  displays  and  microprocessors  do  not  qualify 
for  PM. 

Electromechanical  devices  have  more  moving  parts  and,  therefore, 
more  breakdowns;  electronic  devices,  with  less  moving  parts,  have  less 
breakdowns  so  that  it  becomes  uneconomical  to  have  a  preventive 
maintenance  program  for  them. 

•  Any  device,  however,  left  in  a  totally  dormant  and  inactive  state  for  extended 
periods  should  be  examined  for  corrosion,  dirt  and  free  movement  of  normally 
moving  parts  before  it  is  activated. 

Under  a  maintenance  contract,  PM  will  be  performed  on  a  regularly 
scheduled  basis  dictated  by  the  environment. 
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Parts  within  an  electromechanical  device  that  tend  to  be  in  constant 
nnotion,  or  are  constantly  being  abused  by  a  moving  part  such  as  head 
pads,  belts,  gears,  reed  switches,  or  microswitches,  are  normally 
examined  and  replaced  routinely  during  each  PM.  Most  of  these  parts 
are  considered  expendable. 

The  device  is  normally  thoroughly  cleaned,  vacuumed  and  lubricated 
where  necessary. 

Filters  are  either  replaced  or  cleaned  depending  on  the  type. 

Magnetic  heads  in  tape  drives  and  disk  units  are  also  cleaned. 
Occasionally,  heads  can  become  pitted  as  the  result  of  abrasive 
material  collecting  directly  on  the  head,  or  from  scratched  or  damaged 
media  (tape  or  disk).  Under  such  circumstances,  these  heads  require 
burnishing  and  calibrating.  It  is  recommended  that  these  units  be 
returned  to  the  vendor  for  repair  and  alignment. 

•  Attention  to  the  aesthetics  of  a  device  is  also  included  in  a  good  PM  routine. 
Included  are  such  things  as: 

Cleaning  CRT  screens. 

Video  adjustments. 

Cleaning  keyboards. 

Cleaning  cabinets. 

Replacing  CRTs  with  phosphor  burns  at  a  nominal  cost. 

•  A  well  defined  and  disciplined  PM  program  will  prolong  equipment  life  and 
contribute  significantly  to  the  system  availability. 
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•  On-site  maintenance  can  be  performed  by  either  the  vendor  or  user.  This  is 
usually  dictated  by  the  system  configuration  and  the  user's  ability  to  perform 
maintenance. 

Sophisticated  total  electronic  processing  control  systems  are  usually 
maintained  by  the  vendor's  field  engineering  staff. 

•  This  can  be  handled  in  one  of  three  ways. 

A  maintenance  contract  committing  a  resident  field  engineer  to  be 
dedicated  to  the  user. 

A  maintenance  contract  committing  a  field  engineer  to  arrive  on-site 
within  a  specified  time,  usually  four  hours  from  receipt  of  the  service 
call. 

A  Time  and  Materials  (T&M)  contract  that  simply  places  a  customer  in 
a  queue  for  service.  The  vendor  is  not  obligated  to  respond  in  a 
specified  timeframe.  Customers  with  maintenance  contracts  have 
priority. 

•  Facility  electricians  working  for  the  user  can  be  adequately  trained  to 
maintain  less  complicated  systems  consisting  of  programmable  logic  control- 
lers (PLCs)  and  related  sensors. 

Vendors  of  simpler  systems,  consisting  of  nothing  more  than  PLCs,  do 
not  usually  offer  maintenance  contracts.  It  is  the  consensus  that 
customers  can  be  trained  adequately  to  perform  their  own  maintenance. 

Since  modular  design  is  commonplace  with  today's  hardware  architec- 
ture, the  maintenance  philosophy  is  to  isolate  the  failed  module/com- 
ponent, rem.ove  it,  and  replace  it  with  a  known  working  module/com- 
ponent. 
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•  Vendors,  in  general,  maintain  a  respectable  degree  of  sensitivity  to  customer 
service  needs  under  any  circumstances. 

•  Off-site  maintenance  generally  refers  to  repairs  performed  at  the  vendor's 
repair  depot.  Vendors  may  maintain  one  or  more  strategically  located  depots. 
Their  location  is  usually  dictated  by  two  important  factors  -  equipment  | 
concentration  and  transportation. 

In  the  case  of  multiple  depot  locations  supporting  a  large  product  line 
variety,  it  is  conceivable  that  each  one  will  have  different  levels  of 
maintenance  responsibility.  That  is,  a  depot  may  specialize  in  a 
specific  product  requiring  special  handling. 

This  specialization  provides  quality  repair  due  to  the  repetitive  nature 
of  its  support  function,  excellent  quality  control,  and  faster  turnaround 
time  for  local  customers,  as  well  as  customers  and  FEs  elsewhere. 

Under  any  circumstances,  the  depot  located  at  vendor  headquarters  is 
capable  of  total  repair  support  for  the  customer. 

Remote  diagnostic  capability  for  hardware/software  problem  deter- 
mination in  total  EPC  systems  offers  another  alternative  under  certain 
conditions. 

•  Labor  costs  rising  at  an  annual  rate  of  12%  to  15%  force  field  engineering 
organizations  to  demonstrate  their  ability  to  function  profitably.  The  commit- 
ment to  implement  cost  controls,  increase  the  field  engineer's  productivity, 
and  maintain  the  image  of  being  a  viable  organization  is  a  challenge  of  large 
dimension. 

The  field  engineer's  productivity  is  increased  through  effective  utiliza- 
tion. This  is  accomplished  through  proper  training  in  all  areas  of 
responsibility  including  hardware  troubleshooting  techniques  and 
communication  skills  dealing  with  the  customers  and  support  personnel. 
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The  emphasis  on  communications  will  also  reduce  the  "repeat  call"  and 
"no  fault  found"  syndrome. 

Training  programs  for  both  the  vendor  and  customer  will  constantly  be 
reviewed  to  complement  new  products. 

A  well  defined  escalation  process  for  both  the  vendor  and  customer  will  also 
expedite  effective  and  timely  support. 

Improved  equipment  design  and  reliability  will  improve  system  availability 
while  decreasing  the  MTBF. 

Backup  support  specialists  are  available  to  vendor  field  personnel,  as  well  as 
the  customer. 

Consideration  is  also  given  to  the  removal  of  chronic  devices  detrimental  to 
customer  system  availability  and  field  engineering  productivity.  These  devices 
are  usually  targets  for  a  thorough  overhaul  or  scrap. 

Emphasis  on  modular  design  of  hardware  and  software  impacts  favorably  on 
Mean  Time  To  Repair  (MTTR).  That  is,  the  partitioning  of  hardware  and  some 
software  components  permits  the  rapid  replacement  of  a  failed  component, 
once  it  has  been  identified. 

Absolute  control  of  inventory  requirements  is  another  major  cost  considera- 
tion. Economies  of  scale  are  achieved  by  maintaining  an  inventory  of  parts 
with  the  lowest  MTBF;  in  other  words,  the  most  common  and  frequently 
failing  components. 

Interchangeable  parts  can  be  readily  identified  via  a  cross-reference  chart 
scheme.  This  expedites  repair  whenever  a  direct  replacement  is  unavailable. 
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Documentation  is  nornnaily  included  in  the  sale  of  a  device  or  system.  The 
level  of  documentation  is  consistent  with  the  customer  need  and  supports  both 
the  hardware  and  software. 

Manuals  written  specifically  for  customer  use  adequately  describe  the 
theory  of  operation  for  each  functional  area  of  the  device  or  system. 
These  manuals  include  block  diagrams  and  logic  diagrams. 

Maintenance  manuals  are  not  usually  provided  for  electronic  equipment, 
but  are  provided  for  electromechanical  devices.  These  manuals  provide 
considerable  detail  of  all  mechanical  and  electronic  circuits.  Elec- 
tronic equipment  circuitry  is  much  larger  in  scope  due  to  packaging  and 
would  require  extensive  maintenance  documentation. 

Software  reference  manuals  describing  the  language  used  and  assembly- 
level  programming  assume  that  the  customer  is  familiar  with  these 
topics. 

Spare  parts  management  is  of  little  concern  for  customers  having  a  mainte- 
nance contract.  Such  contracts  normally  obligate  the  vendor  to  total  logistics 
responsibility.  Customers  without  a  maintenance  contract  must  consider 
purchasing  a  recommended  list  of  spares. 

The  level  of  supporting  inventory  is  determined  by  usage.  This 
information  is  gathered  from  historical  data  depicting  all  details  of 
equipment  performance. 

Individual  customer  apprehensions  will  also  impact  the  type  and  level  of 
required  inventory. 

Recommended  spares  usually  represent  10%  of  the  equipment  purchase 
price. 
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Obsolescence  in  the  comnnercial  market  is  not  a  major  concern  since 
most  vendors  will  support  the  product  for  at  least  5  to  I  0  years.  Should 
a  customer  retain  the  equipment  beyond  this  period,  parts  can  be 
custom  made  from  the  original  artwork,  albeit  at  great  expense. 

Parts  are  shipped  in  appropriate  protective  containers.  Vendors 
suggested  that  such  containers  should  be  retained  for  reuse  until  they 
are  no  longer  usable.  However,  if  the  containers  are  made  of  Kraft 
paper  or  the  parts  are  stored  in  corrugated  cardboard  boxes  for  long 
periods  of  time,  the  sulphur  may  seriously  degrade  the  parts. 

•  The  warranty  period  commits  the  vendor  to  total  customer  support.  Beyond 
the  warranty  period,  the  customer  will  negotiate  a  maintenance  contract  or 
assume  a  maintenance  posture. 

Before  system  acceptance,  the  vendor  is  obligated  to  satisfy  final 
checkout  and  acceptance.  This  includes  logistics  support. 

Customers  purchasing  less  complex  systems  using  nothing  more  than 
programmable  controllers  frequently  perform  installation  as  well  as 
maintenance.  The  vendor  and/or  distributor  continues  to  be  liable  for 
startup  problems  within  the  equipment. 

•  Post-acceptance  support  is  available  from  the  vendor  depending  on  the  scope 
of  the  system  configuration  and  the  customer  need.  Vendor  personnel  skill 
levels  will  range  from  a  technical  school  graduate  to  a  BSEE  and,  in  some 
cases,  a  PhD. 

Active  support  of  this  type  Is  usually  available  from  the  vendor  for 
periods  of  five  years  from  purchase  and  installation  on  an  "as  required 
basis." 
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c.  Dormancy 

•  The  operation  of  processing  plants,  which  are  subject  to  shutdown  or  layaway, 
can  be  divided  into  three  phases:  dormant  phase,  startup  phase,  and 
operational  phase. 

In  the  dormant  phase,  the  system  has  been  deactivated  and  is  in 
layaway.  Periodic  tests  of  system  operational  readiness  may  be 
required  to  identify  dormancy  failures  and  initiate  module  repair  or 
replacement. 

The  startup  phase  occurs  during  mobilization  and  may  involve  a 
sequenced  initialization  of  individual  components.  Detailed  diagnostic 
tests  may  be  required  at  all  levels  to  verify  operational  readiness  of 
each  element  within  the  system  before  the  next  step  in  the  sequence 
can  be  taken. 

The  operational  phase  enables  the  continuous  manufacturing  process. 

•  Forty-seven  percent  of  the  vendors  interviewed  had  no  experience  with 
dormancy.  The  remaining  53%  had  limited  experience  with  dormancy  but  had 
definite  ideas  and  recommendations. 

Forty-four  percent  favored  keeping  the  system  in  a  ready  mode.  That 
is,  the  system  should  be  on  and  operating  in  some  mode.  The  mode  of 
operation  would  depend  on  individual  circumstances  such  as: 

If  there  is  a  need  for  further  software  development  or  revision, 
the  operating  mode  could  support  this  effort,  enabling  the 
system  to  be  on  during  the  dormant  period. 

If  there  is  no  need  for  further  software  development,  the 
operating  mode  could  be  a  repetitive  self-test  of  all  critical 
functions. 
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Selected,  or  all,  sensors  could  be  shut  down,  while  the  central 
elements  such  as  the  processor  and  related  programmable 
controllers  and  other  peripheral  devices  continue  to  be  exercised 
in  some  predetermined  mode. 

•  All  vendors  expressed  little  or  no  concern  for  equipment  in  a  dormant  state 
provided  there  was  strict  adherence  to  environmental  control. 

Parameters  of  environmental  control  included  a  temperature  range  of 
-40°F  to  +40°C  and  noncondensing  humidity  of  0%  to  95%. 

•  If  the  dormant  state  of  a  munitions  plant  does  not  lend  itself  to  keeping  a 
system  in  a  ready  (on)  state,  periodic  tests  of  system  elements  may  be  useful 
in  minimizing  plant  startup  time  by  identifying  failures  before  mobilization. 

A  technique  known  as  Test  Interval  Analysis  (TIA),  detailed  in  Exhibits 
V-2  and  V-3,  may  be  useful  in  determining  the  maximum  time  interval 
between  such  tests  for  a  given  level  of  system  reliability. 

This  approach  allows  the  designer  to  perform  an  economic  tradeoff 
between  the  frequency  of  tests  and  the  resultant  reliability  of  the 
system. 

•  The  memory  component  of  most  electronic  process  control  systems  utilizes  a 
backup  battery  power  source.  This  design  feature  is  intended  to  prevent 
destruction  of  the  operating  software  and  any  other  pertinent  data  being  held 
in  memory  during  power  failures. 

The  life  of  these  alternative  power  sources  varies  from  72  hours  to 
three  months. 

Vendors  recommend  that  during  planned  dormant  periods,  the  battery 
packs  be  removed  until  the  system  is  reactivated. 
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EXHIBIT  V-2 


TEST  INTERVAL  ANALYSIS 


•  Prepare  a  list  of  all  system  components /modules. 

•  Estimate  the  failure  rate  of  each  component/module  for 
both  the  operational  and  dormant  modes. 

•  Perform  a  Failure  Modes  and  Effects  Analysis  (FMEA)  as 
itemized  in  Exhibit  V-3. 

•  Construct  an  equation  which  mathematically  represents  the 
reliability  of  the  system.    The  equation  is  a  function  of  failure 
rate,  repair  rate,  test  interval  and  operate  time. 

•  Solve  the  equation,  at  the  specified  reliability  level,  for 
the  test  interval . 
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EXHIBIT  V-3 


FAILURE  MODES  AND  EFFECTS  ANALYSIS  (FMEA) 


• 

Used  to  identify  credible  modes  of  failure  and  to  evaluate 

their  consequence. 

• 

Prepare  a  list  of  all  system  components/modules. 

• 

For  each  component/module,  list  every  possible  credible 

failure  mode. 

• 

For  each  failure  mode,  list  the  following: 

Possible  failure  mechanism 

-    Symptoms  and  local  effects  (including  dependent 

failures) 

-    Method  of  detection. 

Inherent  compensating  provision. 

Effect  on  system. 

Remarks  and  other  effects. 
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If  the  dormant  period  is  expected  to  last  for  extended  periods  of  six  months  or 
more,  selected  components  such  as  the  microprocessor,  disk,  and  other  units 
sensitive  to  environmental  conditions  can  be  packed  away. 

Original  containers  should  be  retained  for  this  reason.  The  components 
should  be  packed  as  they  were  originally  and  again  stored  in  a 
controlled  environment.  The  containers  do  not  permit  any  deviation 
from  the  controlled  environment. 

if  the  original  containers  are  not  available,  heavy  duty  plastic  bags  will 
suffice. 

An  appropriate  covering  should  be  used  as  well  for  the  associated 
remote  components  such  as  programmable  controllers  and  sensors  that 
cannot  be  readily  packed  away. 

The  economics  and  practicality  of  packing  anything  away  should  receive 
careful  consideration,  especially  since  it  entails  disconnecting  equip- 
ment that  will  eventually  require  reinstallation. 

Spares  inventory  should  receive  similar  care. 

When  components  are  unpacked  after  a  lengthy  storage  period,  they  should  be 
physically  examined  for  corrosion  and  condensation. 

This  is  particularly  true  of  all  types  of  male-female  connectors  on  PC 
board  planes  and  magnetic  heads  on  disk  and  tape  components. 

Electromechanical  devices  such  as  printers  should  also  receive  a 
thorough  going  over  for  free  movement  of  heads  and  shafts.  These 
should  receive  PM  before  being  placed  into  operation. 
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Once  the  physical  examination  of  components  is  satisfactorily  com- 
pleted, they  should  be  placed  into  position  and  connected  in  accordance 
with  the  original  configuration. 

The  greatest  caution  must  now  be  exercised  by  strict  adherence  to  the  vendor- 
specified  turn-on  sequence.  Otherwise,  hardware  failures  may  occur. 

Failure  tracking  varies  with  the  maintenance  responsibility.  It  is  a  routine 
responsibility  for  the  vendor  performing  the  maintenance.  Customers 
performing  the  maintenance  tend  to  be  lax. 

Field  engineers  are  required  to  complete  a  service  report  on  every 
action  taken.  This  document,  in  many  cases,  is  a  100%  reporting  tool. 
It  becomes  a  key  ingredient  to  a  management  information  system, 
tracking  not  only  the  FE  productivity  but  the  performance  level  on 
every  system  component.  This  includes,  of  course,  failed  parts. 

This  failure  data  is  utilized  by  the  quality  control  department  to  alert 
production  and  engineering  to  potential  changes,  if  required.  Quality 
control  can  usually  provide  complete  historical  data  on  every  com- 
ponent shipped  to  a  customer. 

This  type  of  feedback  is  key  to  eliminating  repetitive  problems. 

Most  vendors  felt  comfortable  with  a  reactivation  and  diagnostic  frequency  of 
one-month  intervals.  One  vendor  indicated  the  frequency  would  be  dictated  by 
the  type  of  manufacturing  process.  The  more  sophisticated  the  manufacturing 
process,  the  more  frequent  the  reactivation. 

While  all  vendors  expressed  little  or  no  concern  for  dormancy,  they 
favored  keeping  the  system  on  and  in  a  ready  mode.  There  was  also 
general  agreement  that  shutting  a  system  down  for  a  period  would  not 
create  any  catastrophic  failures  at  reactivation,  providing  there  was 
strict  adherence  to  environmental  control. 


Maintaining  a  system  in  an  "on"  or  "ready"  state  offers  four  advantages: 


Systenn  maturity  without  abnormal  degradation. 

Minimum  corrosion  and  condensation. 

Near  immediate  operational  status. 

Increased  confidence  level. 

Maintaining  a  system  in  a  ready  state  necessitates  a  staff  of  at  least 
one  trained  person  to  be  available  for  monitoring  system  performance. 

d.        Maintenance  Philosophy 

Vendor  maintenance  philosophy  is  simply  to  effect  a  repair  in  as  short  a  time 
as  possible.  This  benefits  the  vendor  as  well  as  the  customer.  Since  a  service 
organization  constantly  works  in  a  negative  environment,  it  behooves  the 
vendor  to  maintain  a  low  profile.  Thus,  the  thought,  "fix  it  and  get  out." 

This  philosophy  is  prevalent  for  vendors  who  have  assumed  maintenance 
responsibility.  There  is  little  or  no  control  by  the  vendor  where  a 
customer  has  assumed  the  maintenance  posture. 

Current  modular  design  and  increased  reliability  in  hardware  architec- 
ture have  contributed  to  a  marked  improvement  in  MTTR. 

Maintenance  contracts  generally  commit  to  service  "within  four  hours"  of 
receipt  of  the  service  request  or  "next  half-day."  That  is,  should  the  service 
call  be  placed  in  the  morning,  service  can  be  expected  in  the  afternoon. 
Should  the  call  be  placed  in  the  afternoon,  service  will  arrive  the  following 
morning. 
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Vendors  are  discretionary  about  comnnitting  a  resident  field  engineer, 
as  they  focus  on  productivity  and  costs.  Customers  that  have  such  a 
requirement  can  have  it  included  in  the  maintenance  contract  for  the 
appropriate  costs. 

Customers  in  remote  locations  will  experience  higher  maintenance 
costs  and  longer  response  times. 

•  Built-in  self-diagnostics  permit  the  field  engineer  to  expedite  repair.  Simul- 
taneously, they  simplify  trouble-shooting  for  the  trained  customers.  This 
provides  considerable  improvement  in  equipment  availability. 

Currently,  the  level  of  customer  participation  in  maintenance  is  usually 
poorly  defined.  It  should  be  restricted  to  isolating  the  defective  module 
and  replacing  it.  Customers  should  not  troubleshoot  to  the  component 
level. 

•  Preventive  maintenance  is  performed  routinely  on  a  predetermined  schedule. 
This  is  good  practice  for  equipment  in  use  or  in  a  dormant  state.  It  includes 
running  diagnostic  routines  and  replacing  worn  parts,  cleaning,  and  lubrication. 

« 

•  Hardware  and  software  maintenance  are  generally  performed  by  separate 
personnel.  While  some  cross-training  does  take  place,  it  is  not  yet  prevalent. 

•  Every  customer  service  organization  has  an  escalation  procedure.  The  intent 
is  to  expedite  the  marshalling  of  all  resources  required  to  solve  a  problem. 

•  Vendors  of  the  more  sophisticated  total  electronic  processing  control  systems 
encourage  customers  to  opt  for  a  maintenance  contract.  The  primary  reason 
is  that  the  vendor  is  prepared  with  a  trained  staff,  whereas  the  customer 
usually  cannot  justify  the  related  overhead  and  inventory  expense. 

Maintenance  is  usually  contracted  for  one  year  and  is  renewable. 
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Customers  can  also  opt  for  a  T&M  contract,  where  only  the  costs 
related  to  each  service  call  are  paid. 

Vendors  performing  maintenance  as  their  principal  business  tend  to  display  a 
greater  sense  of  urgency  and  sensitivity  than  do  customers  performing  their 
own  maintenance. 

This  is  because  it  is  not  the  customer's  principal  business. 
Therefore,  it  is  an  inherent  characteristic  of  customers  perform- 
ing their  own  maintenance  to  be  complacent,  except  in  the  case 
of  major  failures. 

Maintenance  personnel  skills  requirements  vary  with  the  vendor  and  system 
configuration.  Manufacturers  of  electronic  processing  control  systems  and 
sophisticated  programmable  controllers  feel  that  advanced  skills  are  required. 

Forty-seven  percent  of  the  vendors  interviewed  required  graduates  of 
technical  schools  trained  by  the  vendor  for  maintenance  of  the  hard- 
ware and  for  reactivation. 

Thirty-three  percent  encouraged  training  of  skilled  customer  facilities 
electricians  for  maintenance  of  the  hardware  and  for  reactivation. 

Thirteen  percent  of  the  responding  vendors  required  their  own  personnel 
to  have  a  BSEE  degree  supported  by  the  vendors'  own  technicians  for 
maintenance  of  the  hardware  and  for  reactivation. 

They  indicated  four  sources  for  such  personnel: 

Undergraduate  technical  schools. 

Technical  schools. 

Competitors. 

Customers. 
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•  All  vendors  offer  training  programs  to  customers.  Hardware  training  programs 
are  structured  to  train  customer-skilled  facilities  electricians  for  simple  and 
logical  trouble  isolation.  There  is  little  or  no  theory  of  operation.  Software 
training  assumes  a  basic  working  knowledge  of  software. 

Twenty  percent  of  the  vendors  mixed  on-the-job  training  with  formal 
training.  The  remaining  vendors  provided  formal  training  only. 

•  All  vendors  had  a  method,  crude  or  otherwise,  for  tracking  the  movement  of 
their  maintenance  personnel.  The  tracking  method  varied  with  the  type  of 
dispatching  scheme. 

Regardless  of  the  tracking  method  used,  the  primary  responsibility  for 
notification  of  a  field  engineer's  whereabouts  belonged  to  the  field 
engineer. 

Computerized  central  dispatching  networks  tracked  an  individual's 
location  in  real-time.  This  method  readily  enables  a  manager  to  divert 
the  best  qualified  field  engineer  nearest  to  a  site  with  a  maintenance 
requirement. 

Satellite  maintenance  offices  without  the  luxury  of  a  sophisticated 
central  dispatch  function  must  depend  on  the  dispatcher's  knowledge  of 
their  whereabouts.  The  dispatcher,  in  turn,  must  depend  upon  the  field 
engineer  to  report  his  location  constantly. 

2.        MAINTENANCE  SERVICE  VENDORS 

a.        General  Climate 

•  System  manufacturing  companies  have  their  own  in-house  field  engineering 
(maintenance)  organizations.   These  FE  organizations  are  increasingly  becom- 
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ing  profit  centers,  as  shown  in  Exhibit  V-4,  and  as  they  do,  the  prime 
difference  between  a  captive  FE  organization  and  a  pure  maintenance  service 
organization  will  be  the  brand  of  equipment  maintained. 

The  captive  organization  maintains  only  one  vendor's  equipment,  while 
the  maintenance  service  organization  maintains  several  brands  of 
equipment. 

•  In  general,  both  kinds  of  organizations  have  similar  personnel,  equipment,  and 
customer  problems. 

•  System  maintenance  is  a  dynamic,  rapidly  growing  area  that  will  blossom  in 
the  1980s  as  a  major  growth  Industry. 

Organizational  structures  are  changing  to  meet  new  needs.  Seventy 
percent  of  the  vendors  created  major  changes  during  1980. 

In  addition  to  structural  changes,  the  number  of  field  engineering 
locations  increased  by  15%. 

•  Recruiting  and  adequate  diagnostic  equipment  were  highest  on  the  list  of  rated 
concerns  by  vendor  FE  organizations,  as  shown  in  Exhibit  V-5. 

These  two  problems  have  or  will  be  experienced  by  the  operating 
contractors  also.  Since  the  contractors  compete  in  the  open  market  for 
personnel,  the  lack  of  skilled  personnel  will  make  recruiting  difficult. 
Test  equipment  for  maintaining  equipment  or  for  testing  spares  Is 
minimal. 

•  All    vendor   field   engineering   organizations   (in   a   previous   INPUT  study) 
averaged  equipment  availability  from  95%  to  96.4%,  as  shown  In  Exhibit  V-6. 
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EXHIBIT  V-4 


PROFIT  CENTER  OR  COST  CENTER 
ORGANIZATION  OF  RESPONDING  VENDORS 
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EXHIBIT  V-5 


VENDOR  RATINGS  OF  PROBLEMS  RELATED  TO  FIELD  SERVICE 


FACTOR  (PERCENT  RATING  HIGH) 


RECRUITING  PIELD  MAINTENANCE 

PERSONNEL  (50%) 


ADEQUATE  DIAGNOSTIC 

EQUIPMENT  (25) 

ADEQUATE  REMOTE  DIAGNOSTIC 

ASSISTANCE  (40) 


PRODUCT  QUALITY  (25) 


SPARE  PARTS  SHORTAGE  (35) 


CUSTOMER  DEMANDS  (30) 


MARKETING  DEMANDS  (30) 


TRAINING  FIELD  MAINTENANCE 
PERSONNEL 


MORALE  OF  MAINTENANCE 
FORCE 


(30) 


REDUCING  LABOR  TURNOVER  (40) 


SALARY  ADMINISTRATION  (10) 


BUDGET  LIMITATIONS  (25) 


(10) 


NUMBER  R£;?C\S£S:  20 
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VENDOR  RESPONSES  TO 
EQUIPMENT  AVAILABILITY 


EQUIPMENT 
TYPE 

AVERAGE 
UPTIME 

MTBF 

MTTR 

AVERAGE  . 
RESPONSE 
TIME 

RESPON- 
DENTS 

PERCENT 

RESPON- 
DENTS 

HOURS 

RESPON- 
DENTS 

HOURS 

RESPON- 
DENTS 

HOURS 

MAINFRAMES 

8 

95.3% 

6 

1,187 

8 

2.56 

7 

1.26 

SMALL 

BUSINESS 

MACHINES 

10 

95% 

7 

2,164 

10 

3.08 

9 

3.17 

MINI- 
COMPUTERS 

6 

96.4% 

5 

2,204 

6 

2.00 

6 

3.18 

PERIPHERALS 

6 

95.9% 

5 

3,548 

6 

1.78 

5 

1.78 

TERMINALS 

8 

95.6% 

6 

5,187 

8 

0.98 

8 

3.48 
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The  highest  nonredundant  mainframe  MTBF  reported  was  2,500  hours,  the 
lowest  was  144  hours.  The  average  MTBF  for  all  mainframe  vendors  came  to 
I ,  I  87  hours. 

Terminals  have  the  greatest  unit  reliability  as  reported  by  vendors  (5,187  hours 
MTBF),  and  the  shortest  repair  time. 

Reliability  of  peripherals  is  distorted  somewhat  by  two  vendors  reporting  over 
6,000  hours  MTBF.  The  norm  is  running  1,500  to  2,500  MTBF  on  peripherals. 

Small  business  machines  are  experiencing  the  same  reliability  as  their  fore- 
bears, the  minicomputers,  with  the  average  MTBF  reported  as  2,164  hours. 
Average  availability  is  pushed  down  to  95%  in  small  business  machines  due  to 
the  slower  turnaround  of  over  six  hours. 

The  "average  uptime"  column  in  Exhibit  V-6  is  an  average  of  averages  which 
will  not  reconcile  against  average  MTBF  and  average  outages.  Response  times 
plus  repair  times  do  not  add  up  to  outage,  which  must  be  used  to  calculate 
availability. 

All  20  vendors  (from  a  previous  study)  reported  that  field  engineers  replace 
boards  on-site  to  effect  repairs,  and  that  they  will  continue  this  practice  in 
I  984. 

A  greater  number  of  respondents  (II  of  20)  indicated  that  their  field 
engineers  will  be  replacing  units  on-site  in  1984. 

Only  four  of  20  respondents  have  their  field  engineers  replace  com- 
ponents on-site,  but  six  vendors  expect  field  technicians  to  replace 
components  on-site  by  1984. 

Ninety  percent  of  respondent  vendors  return  most  boards  and  units  to  a 
factory  for  repairs.  • 
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One-half  of  fhe  vendors  surveyed  repaired  some  boards  and  units  at  a  I 
regional  depot. 

Only  10%  of  the  respondents  repaired  boards  and  subassemblies  on-site. 

•  Most  system  support  centers  are  transparent  to  customers  as  reported  by 
vendors,  with  only  two  vendors  giving  direct  assistance  to  users  from  the 
support  centers. 

Although  I  I  vendors  (55%)  reported  system  support  centers  in  existence 

for  up  to  I  I  years,  the  concept  of  direct  user  access  is  relatively  new.  , 

Seven  of  the  eleven  system  support  centers  are  set  up  to  provide 

software  support  to  the  field.    This  presents  an  opportunity  to  create  ' 

more  direct  lines  of  communication  with  users  as  resistance  to  the  , 

I 

concept  recedes.  | 

Vendors  and  users  alike  seem  to  feel  that  more  selling  and  proving  of 
the  benefits  of  direct  user  interface  with  system  support  centers  are  in 
order. 

i 

Users  have  indicated  their  strong  feelings  about  losing  the 
"personal  touch"  and  about  paying  for  maintenance  to  be  per- 
formed for  them.  I 

i 

Vendors  feel  a  level  of  user  resentment  and  continue  where 

possible  to  interface  through  a  local  maintenance  person  when  I 

providing  assistance. 

I 

•  For  vendors  with  field  forces  greater  than  200,  system  support  centers  use 
from  1%  to  5%  of  available  personnel.  The  two  smallest  vendors  interviewed 
placed  approximately  one-third  of  their  force  in  the  centers,  most  likely 
reflecting  a  desire  to  get  greater  client  coverage  from  a  limited  number  of 
people. 
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•  None  of  the  vendors  reported  an  increase  in  personnel  due  to  the  implementa- 
tion of  system  support  centers. 

Eighty-five  percent  of  respondents  indicated  that  there  was  no  discern- 
ible effect  on  total  personnel  requirements  resulting  from  system 
support  centers. 

Except  for  one  vendor  who  reported  a  personnel  savings  of  84%, 
decreases  in  total  personnel  ran  at  5%. 

•  Four  of  the  nine  respondent  maintenance  vendors  without  system  support 
centers  planned  to  implement  the  concept  between  now  and  1985. 

Two  of  the  four  expect  to  save  15%  in  personnel  requirements  as  a 
result  of  implementation. 

Five  vendors  (25%)  have  no  current  plans  to  implement  system  support 
centers. 

•  High  reliability  technology  at  more  reasonable  rates  is  beginning  to  have  some 
effect  on  the  volume  of  equipment  that  may  be  handled  per  field  engineer,  but 
the  ratios  of  utilization  are  not  impacted. 

•  User  involvement  in  diagnosing  and  solving  problems  offers  some  potentially 
dramatic  relief  at  the  bottleneck,  but  draws  little  attention  from  vendors. 
INPUT  forecasts  an  increase  in  the  level  of  user  involvement  in  maintenance 
as  IBM  and  other  vendors  promote  the  concept. 

Customer  involvement  in  maintenance  and  diagnostics  will  elevate  the 
importance  of  remote  technical  support  techniques  as  customer 
personnel  are  expected  to  interface  and  implement  solutions  suggested 
by  computer  vendors. 
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The  inevitable  evolution  of  user  involvennent  will  place  a  new  burden  on 
the  training  department  to  provide  the  field  with  well-documented 
training  aids  for  users. 

•  Half  the  vendor  respondents  consider  improvement  in  product  price/perform- 
ance ratios  to  have  significant  impact  on  maintenance  techniques,  as  shown  in 
Exhibit  V-7. 

Maintenance  costs  become  much  more  visible  to  the  user  as  price/ 
performance  improves. 

A  selling  point  suggestion  is  to  relate  maintenance  prices  to  perform- 
ance (throughput)  instead  of  units  or  unit  prices. 

•  Vendors  perceive  technological  advances  to  have  medium  to  high  levels  of 
impact  on  maintenance.  ^ 

Upgraded  training  requirements  become  more  frequent  with  new 
technologies. 

Higher  reliability  with  less  frequent  failures  means  slower  learning 
curves  and  longer  mean  times  to  repair. 

Better  price/performance  creates  higher  visibility  to  users  of  increased 
maintenance  ratios. 

•  Distributed  data  processing  (DDP)  was  rated  highly  as  having  an  impact  on 
maintenance  techniques  by  k5%  of  respondent  vendors  (from  a  previous  study), 
as  shown  in  Exhibit  V-7. 

The  dispersed  characteristic  of  DDP  is  causing  some  concern  about  the 
focal  points  of  service. 
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EXHIBIT  V-7 

VENDOR  RESPONSES  ON  IMPACT  OF 
FACTORS  RELEVANT  TO  MAINTENANCE  TECHNIQUES 


FACTOR  (PERCENT  RATING  HIGH) 

RISING  LABOR  COSTS 

INCREASED  PRODUCT  PRICE 
PERFORMANCE 


DISTRIBUTED  DATA 
PROCESSING 


USER/VENDOR  COOPERATIVELY 
TESTING  EQUIPMENT 


(80%; 


(50) 


ADVANCES  IN  TECHNOLOGY  (45) 


(45) 


BUILT-IN  DIAGNOSTICS  (25) 


REMOTE  DIAGNOSTICS  (30) 


MULTIFUNCTION  EQUIPMENT  (30) 


(15) 


USER  PERFORMING  OWN 

MAINTENANCE  (10) 


HOME/PERSONAL  COMPUTERS  (0) 


1  2  3  4  5 

AVERAGE  RATING  (SCALE:  1-5) 

NUMBER  RESPONSES:  20 
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Innovations  like  remote  diagnostics  will  most  likely  be  pushed  to  the 
foreground  by  DDP  maintenance  vendors  over  the  next  few  years. 

b.        Laygway  Approaches 

Maintenance  service  vendors  have  had  little  experience  with  dormant  systems. 

Their  main  function  is  to  keep  systems  running. 

They  have  had  some  experience  with  storage  of  spare  parts  for  varying  periods 
of  time  up  to  about  three  years. 

No  significant  degradation  occurred  to  belts,  wheels,  gaskets,  seals, 
etc.  during  this  short  period.  These  items  were  stored  in  sealed  plastic 
bags. 

Those  electronic  parts  such  as  boards  and  iCs,  which  passed  the  initial 
testing  that  was  performed  when  they  were  received,  survived  two  to 
three  years  of  storage  with  no  signs  of  corrosion  that  might  have 
affected  their  performance.  These  parts  were  kept  in  the  original 
shipping  container  or  in  sealed  plastic  bags. 

The  service  vendors  suggest  that  systems  be  left  "on"  in  some  idling  mode 
when  they  are  not  in  use. 

When  reminded  that  the  period  under  discussion  could  be  I  5  to  20  years 
long,  the  approach  was  modified.  Systems  stored  for  that  period  would 
have  to  be  cycled  periodically.  It  would  be  best  if  the  system  were  left 
"on"  in  an  idling  mode  for  about  one  month  out  of  each  year  or  two  of 
dormancy. 

it  must  be  remembered  that  this  suggestion  is  not  based  upon  "hard" 
data  but  upon  a  "feel"  for  the  situation  coming  from  years  of  mainte- 
nance experience. 


The  service  vendors  feel  that  the  reactivation  of  the  system  at  annual 
intervals  could  serve  as  the  preventive  maintenance  effort. 

Belts  and  wheels  that  have  developed  sagging  or  bumps  due  to  storage 
in  a  particular  position  should  be  replaced  before  the  unit  is  powered 
up. 

All  equipment  should  be  eye-checked  very  carefully  for  loose  boards, 
loose  chips,  and  any  other  anomalies. 

All  units  should  be  powered  up  slowly,  allowing  capacitors  to  reform, 
and  power  supplies  to  warm  up  before  being  asked  for  top  performance. 

OTHER  VENDORS 

Other  types  of  vendor  can  make  a  contribution  in  the  maintenance  area.  Time 
permitted  contacting  a  representative  of  each  of  only  three  types  of  vendor. 
Additional  tasks  should  be  undertaken  to: 

Find  other  vendors  in  these  specific  fields. 

Find  other  specific  industries  that  can  make  contributions  toward  the 
long-term  maintainability  of  these  systems. 

Two  of  the  three  vendor's  products  or  services  deal  with  maintenance  and  will 
be  discussed  here. 

The  third  vendor's  endeavors  are  in  the  training  field  and  will  be 
discussed  in  Chapter  Vll. 

Anticorrosive  coatings  have  been  used  by  the  military  for  long-term  protection 
of  a  variety  of  equipment  for  many  years.  The  type  of  coatings  used  usually 
required  extensive  de-coating  work  before  the  equipment  could  be  used. 
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Such  coatings  are  not  very  useful  when  storing  electronic  components 
due  to  their  small  size  and  the  danger  of  damage  when  handled,  both  in 
applying  the  coating  and  removing  it. 

•         Anticorrosive  vapors  have  been  used  to  protect  electrical  components  while  in 
use  and  in  storage. 

A  potentially  useful  new  product,  introduced  in  1978,  called  ZERUST®, 
is  manufactured  by: 

Northern  Instruments  Corporation 
6680  N.  Highway  49 
Lino  Lakes,  MN  55014 
612-784-1250 

The  ZERUST®  user  that  was  interviewed  found  the  product  to  be  very 
effective  for  use  in  storing  metal  construction  materials  used  in  nuclear 
power  plants. 

The  user  stated  that  Northern  Instruments'  claims  are  understated,  that 
Northern  Instruments  was  technically  honest  and  very  cooperative 
during  the  user's  testing  program. 

Tests  related  to  ZERUST® 's  use  with  integrated  circuits  and  other 
electronic  and  electrical  components  are  included  as  Appendix  E. 

INPUT  has  seen  no  tests  to  indicate  that  the  molecular  vapor  coating  on 
the  IC  pins  can  be  penetrated  by  the  extremely  low  current  used  in 
modern  microcomputers  and  other  !Cs.  If  the  coating  cannot  be 
penetrated  when  the  system  is  reactivated  after  a  long  inactive  period, 
then  the  coating  will  have  to  be  physically  removed,  an  undesirable 
activity. 
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ZERUST"®  comes  in  vapor  capsule  form.  The  capsule  Is  placed  in  an 
enclosure  that  is  circumscribed  but  not  sealed.  The  vapors  will 
permeate  and  protect  all  metal  surfaces. 

ZERUST®  is  also  available  as  a  plastic  bag.  The  bag  has  an  outer 
impermeable  layer  and  a  permeable  inner  layer.  Between  the  layers  is 
the  protective  vaporizing  compound,  as  shown  in  Exhibit  V-8.  These 
bags  are  useful  for  storing  spare  parts  and  subassemblies.  Very  large 
bags  can  be  used  over  whole  cabinets. 

There  are  other  anticorrosive  vapor  compounds  available,  but  non- 
exhaustive  research  indicates  that  ZERUST®  is  a  superior  protector. 

•  There  are  companies  that  specialize  in  the  repair  of  electronic  components. 
Some  of  these  companies  will  produce  boards  for  short  production  runs  if  the 
boards  are  no  longer  available  from  the  manufacturer. 

These  companies  do  not  provide  maintenance,  they  provide  only  repair 
services. 

•  The  largest  of  these  independent  companies 

PSI  Electronics,  Inc. 
I  1900  Mayfield 
Livonia,  Ml  48150 

has  five  service  centers  across  the  country. 

PSI  will  store  a  set  of  board  drawings  so  that  repairs  can  be  completed 
in  minimum  time. 

PSI  will  participate  in  a  parts  exchange  program  in  a  crisis.  It  offers  a 
six-month  operating  guarantee. 
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EXHIBIT  V-8 


ZERUST®  ANTICORROSIVE  PLASTIC  BAG 


IMPERMEABLE 
PLASTIC  LAYER 


PERMEABLE 
PLASTIC  LAYER 


INSIDE  OF  BAG 
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PSI  specializes  in  the  repair  and  conversion  of  obsolete  components. 

PSl  has  competitors.  The  major  third-party  maintenance  companies  use 
other  repair  and  parts  production  shops  when  necessary.  In  the  short 
time  spent  researching  these  companies,  PS!  was  the  only  nationwide 
firm  uncovered.  Additional  detailed  research  is  required  in  this  area. 

C.       USER  ANALYSIS 

I.  SYSTEMS 

•  The  users  interviewed  used  electronic  systems  in  a  wide  variety  of  control 
applications,  as  shown  in  Exhibit  V-9. 

The  processes  ranged  from  control  of  a  single  item  such  as  electric 
power  to  the  complete  control  of  an  entire  film  production  line. 

•  The  systems  about  which  information  was  gathered  included  15-  to  20-year  old 
computers,  as  well  as  some  currently  being  installed. 

The  major  process  control  system  manufacturers  were  represented,  as 
shown  in  Exhibit  V-IO. 

The  newly  installed  systems  in  all  cases  were  replacing  older  systems. 
Existing  systems  were  frequently  undergoing  enhancements  while  they 
were  still  controlling  the  process.  The  addition  of  more  control  points 
(sensors  and  actuators)  was  the  usual  form  of  expansion.  Programmable 
controllers  were  being  added  to  control  the  new  sensors  and  actuators 
in  order  to  lessen  the  burden  on  the  procesor.  This  prolongs  the  life  of 
the  control  processing  system. 
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EXHIBIT  V-9 


TYPICAL  USERS  INTERVIEWED 


INDUSTRY 

APPLICATIONS 

CAMERA  AND  FILM  MANUFACTURING 

AUTOMATIC  FILM  PRODUCTION 

CHEMICAL  PRODUCTION 

SULPHURIC  ACID  BURNER  CONTROL 
TETRAMETHYL  LEAD  MANUFACTURING 

INSTRUMENT  MANUFACTURING 

CHEMICAL  PLATING  PROCESS 

PETROLEUM  REFINING 

CRUDE  OIL  PROCESSING 
CHEMICAL  MANUFACTURING 

PHARMACEUTICAL  MANUFACTURING 

GELATIN  CAPSULE  PRODUCTION 

POWER  UTILITY 

GENERATOR  DEMAND  CONTROL 
PLANT  MONITORING 

STEEL  PRODUCTION 

ELECTRIC  POWER  DEMAND 
MONITORING  AND  CONTROL 

CONTINUOUS  CASTER  FURNACE 
CONTROL 
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EXHIBIT  V-10 


PROCESS  CONTROL  SYSTEMS  AT  USERS'  SITES 


VENDOR 

SYSTEM 

ACCUREX 

PROGRAMMABLE  CONTROLLERS 

ANALOG  DEVICES 

MAC  SYM  2 

BECKMAN  INSTRUMENTS 

MV8000 

CONTROL  DATA  CORPORATION 

CDC  1700 

DIGITAL  EQUIPMENT 
CORPORATION 

DEC  11  SERIES,  DEC  LSl-ll 

FOXBORO 

PCP-88 

GENERAL  ELECTRIC 

GE  3010,  4020,  LOGITROL 

HEWLETT  PACKARD 

HP  1000 

HONEYWELL 

H316,  H716,  H4500 

IBM 

IBM  1800 

LEEDS  AND  NORTHRUP 

L&N  3000 

LOCKHEED 

MAC  16 

MODULAR  COMPUTER 
CORPORATION 

MODCOMP  II 

MOD  I  CON 

PROGRAMMABLE  CONTROLLERS 

TAYLOR  INSTRUMENTS 

MOD  3 

VARIAN 

620 

WESTINGHOUSE 

P250 
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This  increased  distribution  of  control  and  the  use  of  a  hierarchy  of 
systenns  tends  to  increase  the  MTSF  without  having  redundant  hardware 
throughout  the  system. 

The  largest  system  controlled  the  manufacture  of  photographic  film.  Current- 
ly, 8,000  1/0  points  are  monitored  and  controlled  using  about  nine  systems. 
The  goal  is  to  monitor  and  control  about  30,000  I/O  points,  then  raw  materials 
will  enter  at  one  end  of  the  plant,  and  packaged  film  in  a  sealed  box  will  exit 
at  the  other  end. 

It  is  very  curious  that  systems  were  installed  at  the  end  or  beginning  of  a 
decade.  Only  two  within  the  small  sample  were  installed  in  the  middle  of  a 
decade. 

Year  Number  of  Systems 

1958-1961  2 

1968-1969  5 

1973-1975  2 

1978-1980  6 

Systems  generally  were  operating  within  a  comparatively  short  time 
after  procurement.  The  longest  cycle  was  three  years,  with  most 
systems  operating  within  two  years. 

SOFTWARE 

Users  formed  a  symbiotic  relationship  with  the  primary  EPC  system  vendor 
with  users  being  actively  involved  in  all  phases  of  the  design,  equipment 
selection,  installation  and  especially  software  efforts. 
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Vendors  supplied  the  operating  system,  but  the  user  in  conjunction  with 
the  vendor  usually  programmed  the  application. 

One  user  collaborated  with  an  outside  software  house  for  custom 
software. 

One  user  who  very  strongly  believes  in  giving  total  maintenance 
responsibility  to  the  vendor,  accepted  the  vendor's  special  process 
control  language  in  order  to  discourage  custom  software  and  ensure  the 
vendor  would  maintain  it. 

Software  documentation  is  felt  to  be  generally  complete  and  up-to-date  if  the 
vendor  supplied  the  software. 

When  the  user  programmed  the  custom  applications,  the  quality  of  the 
documentation  was  a  function  of  the  documentary  ability  of  the 
language  and  language  processor  used. 

SYSTEM  BACKUP 

Seven  users  had  backup  systems  that  were  redundant  and,  therefore,  were 
ready  to  go  in  case  of  failure  of  the  primary  system. 

The  three  users  that  had  only  manual  backup  systems  all  expressed  a 
fear  that  the  manual  system  skills  will  be  lost  soon  because  the  EPC 
system  is  so  reliable. 

Analog  backup  systems  represented  a  halfway  point  between  computer 
and  manual  controls. 

One  user  is  relying  on  the  self-diagnostic  capability  of  the  CPU  to 
provide  the  reliability  necessary,  and  backup  is  limited  to  disk  drives 
and  interface  controllers. 
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•  Programs  are  stored  generally  on  nnagnetic  media  and  are  kept  in  multiple 
fireproof  locations. 

Few  users  kept  up-to-date  copies  of  the  program  on  fixed  media  such  as 
paper  tape  or  punched  cards  until  the  programs  had  run  long  enough 
with  few  changes  to  assure  people  that  new  cards  or  paper  would  not 
have  to  be  produced  regularly. 

•  Fail  safe  considerations  varied  considerably  depending  upon  the  danger  to 
people  at  the  plant  if  the  process  "ran  away." 

Danger  monitoring  systems  are  rarely  designed  for  worst  case  because  a 
considerable  quantity  of  extra  equipment  is  involved.  The  economics 
dictate  a  certain  amount  of  risk-taking. 

in  one  case  the  extra  equipment  has  been  dismantled  because  the 
system  has  proved  to  be  very  reliable. 

•  The  CPU  is  usually  wired  to  all  I/O  control  points,  to  the  alarm  system,  and  to 
the  fire  protection  system. 

The  CPU  has  an  uninterruptable  power  supply  and,  in  one  case,  the  CPU 
memory  has  a  70-day  battery  backup. 

k.  SUPPORT 

•  The  best  vendor  support  can  be  obtained  by  restricting  the  installation  to  only 
one  vendor. 

Many  users  became  very  involved  in  their  systems  primarily  to  avoid 
having  to  bring  in  another  vendor. 

While  support  has  been  good,  some  users  feel  the  most  consistent 
support  can  only  come  from  within  the  company  and,  therefore,  they 
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have  become  almost  totally  self-sufficient.  These  users  still  rely 
heavily  on  their  single  vendor  for  all  types  of  support,  including  hard  to 
find  competent  people. 

Support  contracts  where  both  parties  are  fully  aware  of  each  other's 
responsibilities  is  a  commonly  found  approach. 

The  older  the  system,  the  more  wary  the  user,  and  the  more  the  user 
depends  upon  his  own  resources.  Also,  over  the  years  this  type  of  user 
has  seen  many  vendors  and  may  now  have  expanded  the  initial  process 
control  system  so  that  many  vendors  are  or  have  been  involved.  Much 
finger-pointing  can  be  avoided  and  time  saved  by  building  a  strong  in- 
house  capability  when  a  multivendor  situation  exists. 

RELIABILITY 

The  experience  of  the  respondents  can  generally  be  classed  as  very  good. 
Systems  are  running  well  especially  after  the  initial  burn-in  period.  Uptimes 
of  99.5%  and  99.8%  were  reported  for  the  CPU.  Some  comments  from  the 
users  illustrate  their  confidence  in  their  systems: 

"First  installation  in  1978  and  have  had  two  hardware  failures  since." 

"Hardware  performs  very  well." 

"No  startup  problems  and  no  major  failures." 

"Very  good  reliability  history.  We  no  longer  keep  a  failure  log  -  it 
gathered  dust." 

Not  everyone  was  so  happy.  Peripherals  presented  problems  from  the  start  on 
some  systems. 

"Selectric  printer  gave  most  problems." 
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"Nixie  tubes  burn  out  after  a  few  years." 

"Peripherals  and  analog  scanner  gave  us  plenty  of  problems." 

"CPU  is  up  99.5%  of  the  time,  but  the  disk  drives  always  seem  to  be 
down." 

A  daily  log  is  used  as  a  failure  report  by  most  of  the  users.  Since  most  users 
have  maintenance  contracts  with  vendors,  it  is  the  vendor's  responsibility  to  do 
the  more  serious  reporting  and  tracking  of  failures. 

One  user  gave  up  the  failure  reporting  system  ten  years  ago  because 
"We  became  recordkeepers  instead  of  engineers." 

Two  users  completed  the  feedback  loop  and  analyzed  the  failure  reports 
in  meetings  between  vendor  and  user  managements.  The  information 
was  used  in  decisions  on  equipment  replacement. 

Users  do  not  concern  themselves  with  MTBF,  MTTR,  and  other  system 
statistics.  If  the  vendor  has  maintenance  responsibility,  the  user  just  wants 
the  system  to  run  24  hours  per  day. 

MAINTAINABILITY 

The  maintainability  of  systems  is  not  a  significant  factor  in  the  design  of  EPC 
systems.  However,  the  most  experienced  user  interviewed  did  claim  that 
systems  with  poor  reputations  for  maintainability  would  be  rejected. 

The  most  prevalent  design  feature  that  the  installed  systems  have  to 
ease  the  maintainability  problem  is  built-in  self-diagnostics.  Usually 
the  respondent  was  not  aware  of  how  much  the  self-diagnostic  features 
of  the  system  contributed  toward  its  selection. 
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As  memory  prices  decrease,  more  users  are  adding  sufficient  memory 
enabling  the  use  of  on-line  diagnostic  software  which  continuously  tests 
the  various  components. 

Modular  system  design  permits  the  pulling  of  malfunctioning  com- 
ponents without  shutting  the  system  down  and  replacing  those  com- 
ponents with  a  spare.  The  bad  component  is  repaired  and  replaced  in 
the  system  or  sent  to  spares  inventory. 

•  EPC  systems  are  frequently  crucial  components  of  a  production  process  that 
runs  24  hours  per  day,  365  days  per  year. 

This  expected  performance,  users  feel,  justifies  the  expense  of  a 
maintenance  contract  with  the  vendor  that  calls  for  a  vendor's  field 
engineer  resident  at  the  customer's  site. 

One  user  who  pays  for  a  vendor's  field  engineer  on-site  for  eight 
hours  per  day,  five  days  per  week  is  planning  on  increasing  this 
service  to  24  hours  per  day,  seven  days  per  week  in  spite  of  the 
200%  increase  in  maintenance  cost. 

•  Preventive  maintenance  is  usually  limited  to  peripheral  equipment;  however, 
few  users  will  tolerate  that  fairly  low  level  of  PM  if  they  have  a  resident  FE. 

•  PM  frequency  is  planned  as  a  function  of  the  particular  system  and  the  process 
under  control. 

PM  on  a  per  month  basis  is  the  norm  among  the  interviewed  companies. 

At  one  company's  site,  PM  was  originally  performed  weekly  and 
gradually  set  back  to  monthly,  based  upon  experience.  At  another  site, 
PM  was  scheduled  on  a  quarterly  basis  but  the  system  had  a  chronic 
problem.  Accelerating  maintenance  to  monthly  cleared  up  the  problem. 
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•  The  users  that  had  long-term  maintenance  contracts  with  the  vendors  were  not 
concerned  about  the  vendor's  source  for  personnel.  The  user  was  concerned 
that  the  FEs  sent  to  work  on  the  system  were  trained  not  only  by  the  vendor 
generally  but  in  the  specific  hardware. 

The  most  important  skill  a  maintenance  engineer  can  have  Is  the  ability 
to  troubleshoot  -  to  logically  diagnose.  Repair  can  be  done  by  less 
skilled  people.  Many  users  felt  that  troubleshooting  skills  are  innate, 
and  the  training  provided  merely  enables  the  engineer  to  develop 
specific  talents. 

•  Users  prefer  to  hire  from  within,  but  as  skilled  personnel  become  increasingly 
scarce,  recruiting  policies  change  to  meet  the  situation. 

Users  are  now  looking  for  trainable  people  and  are  putting  extra  dollars 
into  the  development  of  extensive  training  programs.  Packaged  video 
courses  are  increasing  in  popularity  because  it  becomes  economical  to 
train  individuals,  geared  to  the  student's  own  learning  rate. 

User  training  programs  are  frequently  designed  to  bring  the  student  up 
to  the  level  that  the  vendor  will  accept  in  the  vendor's  training 
program. 

•  The  usual  sources  of  personnel  mentioned  by  the  respondents  were: 

Military. 
Competitors. 
Technical  schools. 
Off  the  street. 
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SPARE  PARTS  MANAGEMENT 


Only  one  user  had  a  spare  parts  management  approach  that  was  called  a 
system  even  though  it  was  an  ad  hoc  development.  Some  important  elements 
of  the  approach  are: 

Buy  spares  through  judgement  and  experience.  Do  not  rely  entirely  on 
the  vendor's  list. 

Keep  track  of  the  time  each  part  is  in  the  system.  Replacement  time  is 
very  important.  Parts  should  be  replaced  before  they  go  bad.  For 
example,  electrolytic  capacitors  go  bad  after  ten  years.  They  should  be 
replaced  at  nine  years. 

Keep  track  of  alternative  sources  of  supply. 
Keep  track  of  supply  depots. 

Remember  that  a  dual  system  inherently  has  spares. 
Test  all  spares  as  they  are  received. 

Store  spares  in  an  environmentally  controlled  area  near  the  computer. 
If  possible,  store  in  a  rack  as  if  in  a  system. 

Most  respondents  with  vendor  maintenance  do  not  have  any  spares  inventory 
on  hand  that  they  own.  The  vendor  keeps  and  maintains  the  inventory. 

No  value  level  pattern  could  be  found  among  those  users  that  kept 
inventory  on  hand.  The  values  cited  were: 

$150. 

$500-600  (for  five  systems). 
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$30,000. 


$120,000. 

Fifteen  percent  of  the  installed  value  of  the  system. 
A  spare  system  (in  storage). 

•  The  user  with  the  $150  inventory  does  not  believe  in  tying  up  money  in 
inventory.  The  factors  influencing  this  conservative  decision  were: 

Warranties. 

Maintenance  contract. 

Reliability. 

Self-diagnostic  features. 
System  redundancy. 

•  Storage  conditions  for  spares  varied  widely  from  a  clean  room  with  foam- 
padded  drawers  to  an  ordinary  file  cabinet  on  the  factory  floor. 

Frequently,  the  spares  are  stored  in  the  original  shipping  containers. 

One  user  who  extensively  tests  the  new  spares  began  the  program  on  a 
regular  basis  because  at  one  time  fully  50%  of  the  spares  were 
defective. 

•  Spares  are  not  usually  guaranteed  beyond  three  years,  although  with  proper 
storage,  one  user  claims  parts  have  been  stored  for  20  years. 
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Maintenance  contracts  usually  specify  that  the  vendor  will  retrofit  the  system 
with  the  latest  parts  so  that  the  system  does  not  become  obsolete. 

Users  plan  on  replacing  their  systems  when  vendor  support  and/or  parts 
are  no  longer  available. 

Vendors  have  promised  to  support  the  system  for  a  minimum  of  five 
years,  and  some  vendors  have  promised  ten  years. 

DORMANCY  EXPERIENCES 

Six  of  the  responding  vendors  had  some  occasion  to  turn  their  system  off  from 
the  process  for  varying  periods  of  time.  Since  the  experiences  vary  greatly, 
each  case  is  described  separately  below. 

a.        User  I 

System  I. 

The  system  was  shut  down  about  ten  months  ago.  The  manufacturer  is 
no  longer  in  the  computer  business  and  parts  are  not  available.  The 
system  is  unprotected  and  sits  in  the  middle  of  a  closed  factory 
building.  It  has  been  totally  abandoned.  The  user  will  probably  never 
use  the  system  again. 

System  2. 

The  system  was  disconnected  from  the  process  in  December  1979  due  to 
lack  of  business.  It  has  been  kept  running  all  the  time  because  the  user 
is  afraid  to  turn  if  off.  The  system  is  idling.  The  system  is  dusted  off 
and  occasionally  used  for  debugging  new  programs.  The  system  will 
rem.ain  in  this  state  until  business  picks  up. 
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•  The  longest  dormant  period  was  three  nnonths  due  to  a  decrease  in  production. 
All  units  are  kept  "on"  now  during  dormant  periods.  Earlier  strategy  was  to 
turn  them  off,  but  this  caused  the  operators  to  forget  to  turn  them  on  when 
reactivation  was  necessary.  In  order  to  reduce  the  loss  of  time  due  to 
forgetting  to  turn  the  control  units  back  on,  the  units  are  now  wired  for 
continuous  "on"  operation. 

c.  User  3 

•  The  system  was  powered  down,  but  the  computer  room  atmosphere  was 
maintained.  The  dormant  period  lasted  about  nine  months.  Upon  reactivation, 
the  contacts  on  the  boards  were  corroded  slightly.  Rubbing  the  contacts  and 
reseating  the  boards  cleared  up  the  problems.  Overall  experience  is  con- 
sidered good. 

d.  User  4 

•  The  plant  was  closed  for  six  weeks  and  the  system  just  turned  off.  The 
computer  room  was  kept  cool  and  I/O  sensors  were  stored  carefully.  Reacti- 
vation was  "horrendous"  due  to  user  "sloppiness"  not  because  of  the  system. 
Problems  encountered  included  warped  boards,  misaligned  boards,  dust,  elec- 
trical shorts,  and  moisture. 

e.  User  5 

•  Systems  stored  for  two  years  were  covered  with  plastic  bags  and  kept  in  a 
heated  warehouse.  Magnetic  tapes  and  disks  were  stored  in  a  humidity- 
controlled  area.  Upon  reactivation,  the  units  were  dusted,  vacuumed,  and 
thoroughly  inspected.  They  were  turned  on  according  to  the  vendor's  instruc- 
tions for  new  installations,  and  there  were  no  major  problems.  Power  supplies 
were  the  most  sensitive  component. 
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f.        User  6 

•  Process  was  shut  down  in  January  1981  but  all  control  systems  are  kept 
running,  with  applications  software,  except  for  the  sensors.  The  system  is  still 
in  the  same  state  as  of  this  writing. 
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VI       SPARE  PARTS  MANAGEMENT 


•  This  chapter  corroborates  the  general  information  obtained  from  the  inter- 
views with  information  from  other  INPUT  studies. 

Hard  interview  information  was  sparse  because  the  vendors  felt  that 
their  practices  were  geared  toward  different  goals  than  a  customer 
would  have.  The  users'  interviewees  did  not  have  specific  management 
practices  that  were  codified  such  that  they  could  be  delineated  easily 
for  passing  on  as  advice  to  others. 

Practices  were  ad  hoc  for  the  most  part  among  all  users  interviewed 
including  the  ammunition  plants.  The  ammunition  plant  information 
can  be  found  in  Chapter  IV.  User  and  process  control  vendor  informic- 
tion  is  in  Chapter  V. 

•  Spare  parts  management  should  be  appropriately  addressed  whether  or  not  the 
user  has  a  maintenance  contract.  Maintenance  contracts  currently  obligate 
the  vendor  with  total  logistics  (parts)  responsibility.  Customers  v/ithout  a 
maintenance  contract  must  consider  purchasing  a  recommended  list  of  spares. 

The  goal  of  inventory  management  is  to  satisfy  the  need  for  necessary 
parts  and  supplies,  with  attention  to  control  of  the  inventory  and  its 
costs. 
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Inadequate,  or  worse  yet,  incorrect  inventory  will  result  in  dissatisfac- 
tion due  to  the  weakened  support  capability.  Such  situations  force 
everyone  involved  to  divert  their  attention  end  energy  toward  resolving 
a  parts  situation  that  should  not  have  occurred. 

Inventory  considerations  are  a  major  part  of  the  support  function, 
whether  or  not  the  support  comes  from  the  vendor  or  the  customer. 

Inventory  costs  are  a  major  consideration,  since  inventory  on  hand  is  increas- 
ing in  value  due  to  higher  replacement  costs  and  demand. 

Packaging,  shipping,  handling,  testing  and  repair  can  cost  as  much  as 
25%  of  the  retail  price  for  a  given  part.  Parts  of  less  than  $25  value 
are  normally  expendable. 

If  inventory  levels  permit,  the  first-in-first-out  (FIFO)  method  should 
be  used.  This  will  minimize  the  cost  factor  as  well  as  potential  shelf- 
life  problems. 

Technological  advances  through  continued  research  and  development  will 
impact  the  inventory  process.  Theoretically,  improvements  in  equipment 
relioDility  will  lower  the  required  spares  levels. 

Whenever  a  customer  upgrades  the  system  to  a  later  design,  the 
supporting  inventory  should  be  upgraded  as  well. 

Once  inventory  objectives  are  established,  they  must  receive  full  support  from 
all  responsible  parties. 

The  customer  must  learn  how  long  the  product  will  be  supported  after 
production  terminates.  Armed  with  this  knowledge,  the  customer  can  develop 
long-term  strategy  for  inventory  requirements. 


Individual  customer  apprehensions  over  "recommended"  spares  will  also 
impact  the  type  and  level  of  customer-owned  inventory. 

Recommended  inventory  usually  represents  10%  of  the  equipment 
purchase  price. 

Inventory  purchase  after  production  has  stopped  will  be  expensive  even 
if  drawings  and  tools  are  retained.  Replacing  parts  will  require 
manufacturing  resource  planning  in  a  custom  mode. 

This  would  be  true  also  if  the  customer  had  the  appropriate  artwork  to 
support  manufacturing  at  a  third-party  source. 

The  vendor  has  no  obligation  to  maintain  a  customer's  inventory  to  current 
revision  levels.  One  way  of  being  kept  informed  of  revisions  is  for  the  vendor 
to  place  customers  on  the  mailing  list  for  all  engineering  and/or  field  changes 
for  their  particular  hardware. 

Typically,  not  all  design  changes  will  alter  the  basic  operation  of  a 
device  or  system,  so  it  is  not  necessary  to  upgrade  a  customer's 
inventory  with  each  change. 

Major  changes,  however,  that  will  alter  the  operation  of  a  device  or 
system  compel  the  customer  to  consider  adopting  the  latest  revision  of 
that  particular  subassembly.  This  is  usually  done  at  a  reasonable  cost. 

The  useful  life  of  most  spare  parts  is  at  least  five  years.  Any  investment 
necessary  to  maintain  them  at  current  revision  levels  is  justified,  as  well  as 
time  spent  for  periodic  testing. 

Inventories  should  be  consolidated  as  much  as  possible,  consistent  with 
providing  a  timely  supply.  Elapsed  time  will  be  shortest  when  parts  are  easily 
accessible. 
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Parts  with  the  highest  usage  and  frequent  turnover  should  be  located 
nearest  the  equipnnent  location. 

Parts  with  the  least  usage  and  turnover  can  be  centralized  or  ordered  as 
required.  Vendors  will  usually  respond  within  24  hours. 

Vendor  inventory  management  is  usually  very  sophisticated.  Large  vendors 
track  inventory  in  a  real-time  mode  using  a  computerized  system.  If  a  part  is 
unavailable  locally,  management  is  then  capable  of  locating  it  (nationally)  and 
arranging  its  shipment  immediately. 

Smaller  vendors  without  a  computerized  tracking  network  are  usually  no  more 
than  three  days  (mail)  behind  current  inventory  status.  More  often  than  not, 
they  can  perform  just  as  efficiently  when  locating  a  needed  part  and  shipping. 

Periodically,  vendors  find  it  necessary  to  "shake-out"  inventory  that  has  been 
"accumulated"  by  FEs.  This  is  particularly  true  of  defective  parts. 

During  unusually  busy  service  periods,  it  is  not  uncommon  for  FEs  to 
change  their  priorities  and  delay  the  return  of  defective  parts  to  the 
repair  cycle. 

It  is  critical  that  the  flow  of  these  defective  parts  continue  through  the 
inbound  "pipeline"  so  they  may  be  repaired  and  returned  to  the  outgoing 
"pipeline." 

The  human  element  is  critical  to  any  inventory  control  system. 

Inventory  control  is  a  very  complex  issue,  consisting  of  many  elements  such  as 
correct  part  number  identification,  correct  replacement  (direct)  part  number 
identification,  order  communications,  availability,  integrity  of  the  parts 
performance,  shipping  and  packing,  return  of  the  defective  part  if  not  an 
expendable  item,  and  debit  or  credit  of  inventory  level. 
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•  Vendors,  as  well  as  customers,  must  assume  a  degree  of  risk  for  "out  of  stock" 
items.  Economies  of  scale  simply  do  not  make  it  practical  to  be  in  a  position 
to  fulfill  100%  of  all  needs. 

Consider  that  even  a  success  of  95%  leaves  5%  failures. 

A/B/C  Analysis,  also  known  as  "Tlie  Principle  of  the  Critical  Few  and 
Insignificant  Many,"  "Pareto's  Law,"  and  the  "80-20  Rule,"  helps  direct 
attention  to  the  most  important  items. 

Pareto's  Law  suggests  that,  regardless  of  inventory  ordering  criteria, 
the  distribution  of  parts  on  a  percent  basis  will  remain  the  same. 

An  80%  service  level  can  usually  be  met  with  less  than  20%  of  all  field 
replaceable  parts.  A  90%  level  has  been  observed  with  only  10%  of  the 
possible  replacement  parts  in  stock,  correctly  chosen  in  proper  quant- 
ities. 

•  Once  inventory  requirements  and  levels  are  established  and  satisfied,  vendors 
will  satisfy  internal  needs  and  customer  needs  in  accordance  with  an  opera- 
tional priority  code.  It  would  be  appropriate  for  customers  to  establish  a 
similar  code.  The  intent  is  to  sensitize  the  logistics  function  to  all  situations. 

Typical  operational  priority  codes  are: 

Priority  I  -  Customer  has  experienced  a  catastrophic  failure 
requiring  necessary  parts  immediately.  If  the  part  is  not 
available  locally,  it  must  be  located  and  placed  on  the  next 
available  flight,  while  the  FE  stands  by  to  pick  it  up  and  satisfy 
the  problem. 

Priority  2  -  Customer  equipment  is  down,  but  not  hampering  any 
critical  phase  of  the  operation,  if  a  part  is  not  available  locally, 
it  must  be  located  and  made  available  within  24  hours. 
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Priority  3  -  Customer  equipment  may  or  may  not  be  down. 
There  is,  however,  a  marginal  or  intermittent  problem.  If  the 
part  is  not  available  locally,  it  can  be  ordered,  processed,  and 
shipped  routinely.  Elapsed  time  from  receipt  of  order  is 
typically  five  days. 

•  In  conjunction  with  these  operational  priority  codes,  equipment  and  selected 
parts  are  coded  in  order  to  attach  an  appropriate  degree  of  importance  to  the 
device  or  part.  Priority  I  indicates  the  highest  priority. 

Typical  examples  are  a  microprocessor  would  be  a  5,  a  CPU  board 
within  the  microprocessor  a  4,  a  modem  3,  keyboard  switch  2,  and 
indicator  lamp  I. 

•  The  combination  of  the  operational  priority  code  and  part  rating  allows  a  rapid 
calculation  of  the  action  to  be  taken  in  response  to  any  situation. 

The  rating  number  should  be  assigned  to  each  part  by  maintenance 
management  with  the  concurrence  of  inventory  planning. 

Attention  should  be  paid  first  to  all  5-rated  parts,  then  4s,  3s,  2s  and 
finally  Is.  If  the  inventory  budget  is  restricted,  it  should  be  invested 
first  in  all  the  needed  5-rated  items,  working  down  in  priority  on  lower 
rated  items  until  the  money  supply  is  exhausted.  This  could  mean  that 
no  1  or  2  items  are  stocked;  but  the  effect  on  system  operation  should 
be  minimal. 

•  Flexibility  is  important  to  spare  parts  inventory  management  since  more  than 
one  source  can  often  be  used  to  solve  a  parts  storage  problem.  This  involves 
an  ability  to  determine  where  the  needed  part  might  be  among  several 
different  locations.  In  addition  to  getting  a  part  from  inventory,  the  use  of 
the  next  higher  assembly,  borrowing  from  other  products  the  part  is  used  on, 
or  even  cannibalization  can  provide  emergency  alternatives. 
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•         Inventory  and  nnaintenance  personnel  often  go  to  extreme  efforts  to  get  parts 
necessary  to  repair  the  system. 


Managers  must  be  alert  and  not  allow  these  special  efforts  to  become 
the  norm. 

Emergency  parts  requisitions  exceeding  10%  of  total  requisitions  are  a 
sign  of  poor  inventory  management. 

•  Customers  should  review  fundamental  inventory  planning  parameters  with  the 
vendor.  For  any  given  system,  these  levels  should  complement  the  system 
configuration. 

Although  the  "recommended"  spares  list  is  a  necessary  tool  for  planning 
and  control,  real  life  experiences  should  be  used  to  change  the 
"recommended"  spares  list  accordingly. 

Parts  not  requiring  heavy  usage  can  normally  be  traded. off  for  parts  of 
higher  usage.  Any  difference  in  their  dollar  value  will  be  debited  or 
credited. 

The  major  reason  for  overstocking  parts  is  a  lack  of  confidence  in 
adequate,  timely  resupply. 

Minimumi  downtime  results  if  maintenance  has  the  part  or  can 
get  it  within,  for  example,  one  hour. 

Standardization  is  important.  Not  only  are  fewer  parts  required  and 
costs  lowered,  but  also  the  need  for  training,  tools  and  manuals  is 
reduced.  Program  management  must  ensure  that  tradeoffs  between 
maintenance  requirements  and  those  of  accounting  and  engineering  are 
well  defined  and  understood. 
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Stock  lists  should  be  established  based  on  nearby  similar  systems  and 
special  conditions. 

These  parts  are  identified  by  maintenance  personnel,  as  dis- 
cussed earlier,  and  quantities  are  established  in  cooperation  with 
inventory  planning. 

Cooperative  efforts  among  operating  contractors  with  similar 
systems  should  be  encouraged. 

Maintenance  personnel  often  consider  the  use  of  stock  lists 
extremely  restrictive,  but  experience  shows  that  the  lists  are  a 
necessary  tool  for  planning  and  control. 

•  Carrying  and  holding  costs  include  space,  security,  information,  obsolescence, 
protection  from  damage,  insurance  and  either  the  cost  of  money  invested  in 
inventory  or  the  value  of  forgone  investment  in  other  alternatives. 

Carrying  costs  will  range  from.  20%  for  small,  stable,  low-value  parts  to 
30%  for  electronic  components  that  are  more  valuable  and  subject  to 
obsolescence,  theft  and  damage. 

•  Inventory  classifications  include  working  stock  (on  hand),  pipeline  (inbound  and 
outbound),  safety,  and  loaned. 

Working  stock  (on  hand)  represents  the  stock  in  inventory  and  ready  for 
use. 

Pipeline  parts  are  "in  transit." 

Inbound  -  defective  parts  being  returned  for  repair. 
Outbound  -  repaired  parts  returning  to  inventory. 
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Safety  stocks  are  extra  quantity  kept  as  insurance  against  contin- 
gencies. 

Loaned  stocks  are  those  being  "temporarily"  used  at  another  location. 
They  are  expected  to  be  returned  or  replaced. 

•  Parts  in  inventory  should  be  capitalized  until  they  are  used. 

•  Inventory  for  new  systems  involves  brand  new  equipment,  very  little  history, 
some  correlation  with  prior  products,  reliability  predictions,  some  laboratory 
tests  and,  most  importantly,  people's  best  engineering  experience  and  educated 
judgement. 

Planning  inventory  for  new  systems  requires  input  from  the  vendor's 
engineering,  quality  assurance,  production,  marketing  and  experienced 
field  personnel.  Initial  stock-keeping  unit  (SKU)  identification  should 
be  based  on  a  part's  expected  failure  rate.  Reliability  engineering  can 
assist  with  failure  modes,  effects,  and  criticality  analysis.  Quantities 
are  based  on  the  location  of  stocked  spares,  lead  times  for  resupply, 
essentiality,  and  cost.  Experience  with  comparable,  existing  products  is 
the  best  guide. 

The  need  for  adequate  replacement  parts  as  well  as  trained  personnel, 
tools  and  information  should  force  consolidation  of  early  installations. 

•  Once  operational  experience  is  gained  with  the  new  systems  (generally  at 
about  the  six-month  point),  inventory  of  replacement  parts  may  be  assumed  by 
the  maintenance  group. 

Engineering  changes,  modifications,  and  new  versions  of  equipment 
often  cause  subsystems  to  revert  to  initial  provisioning  techniques. 

•  The  termination  phase  requires  the  acquisition  of  enough  parts  to  support 
service  needs  as  long  as  the  products  remain  in  use. 
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End-of-life  planning  for  parts  must  take  place  when  a  product  will  no 
longer  be  manufactured  or  supported.  Management  must  decide  how 
long  the  system  will  be  operated  after  support  is  terminated.  (Mainte- 
nance management  and  vendor's  spares  analysts  must  then  forecast  part 
demand  during  that  period. 

Additional  purchases  after  support  has  stopped  will  be  very  expensive, 
even  if  drawings  and  tools  are  available,  so  the  end-of-life  spares 
purchases  should  be  made  during  the  last  production  run. 

An  Inventory  control  system  must  provide  accurate  information  at  all  times. 
Knowing  the  location  and  availability  of  a  part  is  critical  to  overall  system 
performance. 

This  information  can  be  maintained  manually,  or  it  can  be  computer- 
based. 

Either  way,  the  discipline  and  competence  of  the  people  maintaining 
the  system  are  crucial. 

A  manual  system  using  a  card  for  each  SKU  is  very  effective. 

Personnel  must  be  trained  and  controlled  to  make  the  appropriate 
entries  whenever  a  part  is  removed  or  added  to  the  inventory. 

Computerized  control  allows  inventory  records  to  be  updated  instantaneously. 
This  will  make  inventory  data  available  in  a  real-time  mode  without  having  to 
conduct  a  physical  search  on  the  premises,  or  elsewhere. 

A  few  fundam.ental  but  critical  techniques  relative  to  the  liandling  of  spares 
inventory  become  paramount.  They  include  dating  and  storage. 

Most  electronic  parts  have  a  shelf  life  of  five  years  or  more;  however,  it  is 
necessary  to  date  them,  especially  if  the  system  is  dormant. 
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•  In  the  case  of  mechanical  assennblies  requiring  some  form  of  lubrication, 
dating  will  support  a  schedule  of  periodic  examination  and  lubrication,  as 
required.  Without  it,  a  frozen  mechanical  assembly  will  prove  devastating  at 
the  time  of  installation. 

•  At  the  time  the  "recommended"  spares  complement  is  purchased,  it  is  assumed 
that  all  the  parts  are  new.  Although  all  parts  have  been  through  a  test  cycle 
and  quality  control,  the  user  should  be  concerned  with  "infant  mortality." 

•  While  it  may  not  be  necessary  to  test  all  parts  in  the  inventory,  a  test  program 
should  be  developed  for  selected  critical  parts. 

•  Examples  of  selected  critical  parts  are  CPU  board,  power  supplies,  print 
heads,  disk  drives,  communications  and  interface  controllers,  and  some  cables. 

•  Once  the  system  is  installed  and  operational,  a  test  program  should  be 
developed  that  would  exercise  these  parts  within  the  system  for  a  period  of 
time.  This  will  satisfy  the  infant  mortality  concern  by  "shaking  out"  marginal 
or  defective  inventory. 

•  Once  this  has  been  satisfied,  the  confidence  level  of  the  inventory  will  be 
attained.  Similar  testing  should  be  done  for  repaired  parts  returning  from  the 
vendor  repair  cycle,  although  there  is  less  concern  for  a  used  part  relative  to 
infant  mortality. 

•  Storage  is  equally  critical.  The  first  consideration  is  the  proximity  of  the 
inventory  to  the  equipment.  This  is  important  if  elapsed  time  is  to  be  kept  to 
a  minimum. 

•  Finally,  environmental  control  for  inventory  is  more  critical  than  it  is  for  an 
operational  system,  considering  that  the  part  may  be  "gathering  dust"  and 
aging  while  in  storage. 
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VI  I  PERSONNEL 


VII  PERSONNEL 


•  Information  concerning  vendor  field  service  personnel  is  deemed  of  concern  to 
ARRCOM  and  the  operating  contractors  due  to: 

The  use  of  vendor  field  service  organizations  at  the  ammunition  plants. 

The  need  for  such  personnel  by  the  operating  contractors  who  must 
compete  in  the  open  market  for  them. 

A.       VENDOR  FIELD  SERVICE  ORGANIZATIONS 

•  Field  service  organizations  are  expected  to  grow  steadily  in  the  coming  years, 
as  shown  in  Exhibit  Vll-I. 

Maintenance  vendors'  growth  is  expected  to  be  greater  than  the  growth 
forecast  by  the  field  engineering  groups  within  system  vendors'  organi- 
zations. 

•  Field  engineering  turnover  has  compounded  recruiting  problems  which  had 
already  become  critical  because  of  growth,  as  shown  in  Exhibit  VII-2. 

Nineteen  of  the  vendors  (from  a  previous  study)  reported  hiring  3,706 
field  engineers  to  replace  2,316  for  a  net  gain  of  1,390. 
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EXHIBIT  VII-1 


EXPECTED  INCREASE  IN  FIELD  SERVICE  ORGANIZATION  SIZE, 

1980-1982-1984 
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EXHIBIT  VII-2 

COMPARISON  OF  FE  NEW  HIRES  AND  SEPARATIONS  -  1979 


6     7     8     9    10    111213    1415    16    17    1819  20 
RESPONDING  COMPANY  NUMBER 
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Because  of  turnover,  more  new  hires  are  required  to  gain  employees  | 
necessary  for  growth. 

The  average  vendor  hired  195  new  field  engineers  to  replace  122  for  a 
gain  of  73. 

Attrition  was  due  to  several  familiar  causes  and  some  new  causes  as 
reported  by  vendors.  Comments,  some  of  which  follow,  ran  76% 
voluntary  resignations  versus  12%  each  for  promotions  and  involuntary 
separations. 

"Wanted  to  leave  the  area." 

"More  money." 

"Career  change." 

"Move  to  warm  climate." 

"Spare  parts  not  available." 

Many  of  these  same  reasons  for  attrition  apply  to  the  operating 
contractor's  maintenance  personnel  also.  In  addition,  the  operating 
contractor  is  maintaining  a  dormant  system  in  many  instances,  and  this 
provides: 

No  challenge  to  maintenance  personnel. 

No  chance  to  do  an  outstanding  job  justifying  a  promotion. 

No  justification  to  request  or  earn  salary  Increases. 

In  summary,  then,  the  operating  contractor's  skilled  maintenance 
personnel  are  in  a  stagnant,  stultifying  situation  facing  an  extreme 
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shortage  of  their  skills  in  a  marketplace  eager  to  pay  top  dollars  for 
their  expertise. 


•  The  trend  toward  trade  schools  as  a  primary  source  of  new  field  engineers  has 
become  even  more  evident  since  1978,  as  shown  in  Exhibit  VII-3. 

Eleven  of  the  respondent  vendors  (55%)  rated  trade  schools  of  great 
importance  in  1980.  In  1978,  only  46%  gave  trade  schools  high  ratings. 

Trade  schools  were  rated  second  in  importance  to  recruiting  from 
competition  in  1978,  but  were  projected  into  first  place  by  1982.  This 
current  survey  finds  trade  schools  already  in  first  place,  with  only  two 
vendors  rating  them  of  little  or  no  importance  in  1985. 

•  The  traditional  source  of  technicians  -  training  the  armed  forces  -  remains  in 
third  place,  with  exactly  50%  of  respondent  vendors  considering  this  source  as 
having  little  or  no  importance.  By  1985,  65%  of  the  vendors  move  the  armed 
forces  into  last  place  by  rating  the  source  low  or  not  applicable. 

•  Recruiting  from  competition,  which  ranked  as  the  number  one  source  of  new 
personnel  in  1978,  has  now  slipped  to  second  place,  with  19  respondents  rating 
competitive  sources  evenly  from  high  to  low  in  importance.  Recruiting  from 
competition  moves  into  a  virtual  tie  with  the  armed  forces  for  last  place  in 
1985,  according  to  respondents. 

•  Respondent  vendors  give  little  importance  in  1980  to  hiring  and  training 
personnel  with  no  technical  background,  but  place  this  source  very  closely 
behind  trade  schools  by  1985.  Seventy-five  percent  of  the  respondent  vendors 
rate  this  source  of  new  field  engineers  as  medium  to  high  in  1985. 

•  Other  sources  of  replacement  or  new-hires  in  field  service  include  four-year 
degree  schools  as  specified  by  one  vendor.  This  vendor  rated  universities  as 
high  in  importance  both  now  and  in  1985,  and  gave  only  low  marks  to  all  other 
sources. 
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EXHIBIT  VII-3 


RATINGS  OF  PRIMARY  SOURCES  OF  NEW  FS  PERSONNEL 


VARIABLE 

YEAR 

1978* 

1980** 

1982* 

1985** 

HIRE  AND  TRAIN 

(NO  TECHNICAL  PRE-TRAINING) 

2.0 

1.8 

2.8 

3.5 

RECRUIT  FROM  COMPETITION 

3.0 

2.9 

2.8 

2.1 

RECRUIT  FROM  INDUSTRIES 

2.3 

2.1 

2.6 

2.4 

TRAIN  DISCHARGED 

ARMED  SERVICES  PERSONNEL 

2.6 

2.5 

2.4 

1.9 

RECRUIT  FROM  OTHER 
FUNCTIONS  WITHIN 
THE  COMPANY 

2.7 

• 

1.9 

2.5 

2.2 

TRADE  SCHOOLS 

2.9 

3.7 

4.0 

4.1 

(SCALE:  1  =  LOW,  5  =  HIGH) 


*1978  &  1982  RATINGS  FROM  "MAINTENANCE  REQUIREMENT  FOR  THE  INFORMATION 
PROCESSING  INDUSTRY  1978-1983";  SURVEY  DONE  IN  1978. 

**1980  AND  1985  FIGURES  FROM  CURRENT  SURVEY 

NUMBER  OF  RESPONSES:  19 
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Another  vendor  rated  universities  as  medium  in  importance  now  and  in 
1985. 

The  third  vendor  who  specified  a  source  other  than  those  listed  said 
that  he  rated  two-year  associate  degree  colleges  medium  in  1980,  but 
very  high  by  I  985. 

Field  engineering  trainees  could  be  hired  into  base  pay  ranges  from  $4.73  per 
hour  ($9,800  per  year)  to  $1,900  per  month  ($22,800  per  year).  The  actual  base 
salaries  paid  to  trainees  by  respondent  vendors  averaged  from  $12,000  to 
$20,220  per  year. 

Base  salary  ranges  for  qualified  field  engineers  on  respondent  vendor  payrolls 
are  from  $12,000  to  $25,800  per  year.  The  actual  salaries  paid  to  qualified 
field  engineers  average  from  $15,000  to  $23,000  per  year. 

Annual  salary  brackets  for  senior  field  engineers  range  from  $13,500  to 
$29,280  per  year,  and  average  from  $17,000  to  $26,000  per  year. 

The  relatively  narrow  range  of  salaries  from  trainee  to  senior  field  engineer 
shows  little  room  for  income  growth  commensurate  with  growth  in  capability; 
this  fact  undoubtedly  contributes  to  turnover. 

Exhibit  VII-4  provides  a  snapshot  of  the  average  salaries  paid  all  field 
engineers  by  respondent  vendors.  The  distribution  of  average  salaries  remains 
consistent  from  trainee  to  senior  in  all  companies  except  numbers  7  and  8. 

Respondent  vendors  reported  increases  in  salary  ranges  from  6%  to  15%  in 
1980,  with  the  average  adjustment  being  9.5%. 

Reasons  given  for  increases  in  salary  ranges  included: 

"Competition." 
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COMPARISON  OF  AVERAGE  SALARIES 


_J  !  !  \  \  !    '    I    I    i  LJ  \  \  L 

6  7  8  9  10  n  12  13  14  15  16  17  18  19  20 
(ND)  (ND)      im)  (ND) 


COMPANY  NUMBER 

ND  =  NO  DATA 
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"Cost  of  living." 
"Increasing  FE  responsibilities." 
"Inflation." 
"Supply  and  demand." 
Reasons  cited  by  vendors  for  individual  increases  in  salaries  included: 
"Merit." 

"Exceptional  performance." 

"Seniority." 

"Promotion." 

"Six-month  reviews  below  mid-range,  annual  reviews  above  mid- 
range." 

It  is  relevant  that  the  inflation  rate  in  1979  was  greater  then  the 
average  salary  increase  reported  by  vendors;  this  can  be  a  contributor 
to  unrest  among  current  field  engineers. 

•         Five  respondent  vendors,  all  with  West  Coast  headquarters,  reported  some 
form  of  incentive  program  for  field  engineers,  including: 

Commissions  on  service  contracts  and  training  courses  sold. 

Quarterly  award  to  "FE  of  the  Quarter,"  one  for  each  region;  one  week 
combined  business  and  pleasure  meeting. 

"Extra  mile"  awards  for  plus  performers. 
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Percentage  of  contract  revenues  allocated  as  bonuses  (two  vendors). 

•  The  absence  of  incentive  programs  at  15  respondent  vendors  often  reflects 
unsuccessful  incentive  programs  tried  in  the  past. 

•  While  commission-oriented  incentive  programs  are  not  germane  to  the  operat- 
ing contractor's  situation,  some  incentive  program  will  be  necessary  in  order 
to  minimize  turnover  rates  of  skilled  troubleshooting  personnel. 

•  Some  operating  contractors  have  commented  upon  the  negative  effect  affirm- 
ative action  programs  have  had  on  their  recruiting  and  training  costs. 

Vendors  also  report  that  recruiting  costs  have  increased  from  5%  to 
30%  due  to  the  need  to  meet  affirmative  action  requirements. 

One  vendor  reported  an  increase  of  30%  in  training  costs  to  meet  the 
guidelines. 

ARRCOM  should  recognize  the  potential  extra  expense  burden  and 
allocate  funds  accordingly. 

B.  TRAINING 


•         Most  of  the  skilled  people  in  the  process  control  field  have  been  trained  by  the 
vendors. 

This  is  especially  true  of  service  personnel  who  may  have  received  their 
initial  electronics  training  in  school  but  received  detailed  technical 
training  by  the  systems  vendor  for  whom  they  work,  or  by  the  vendor  of 
the  system  used  by  the  com.pany  for  whom  they  work. 
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•  Training  techniques,  personnel  trends,  and  the  coming  shortage  of  personnel 
have  been  discussed  in  Chapter  ill. 

A  list  of  companies  providing  training  services,  including  simulators, 
can  be  found  in  the  Appendix  C  listing  of  professional  services 
companies. 

•  User  personnel,  trained  by  the  systems  vendor,  maintain  and  upgrade  their 
skills  as  they  work  with  the  system  on  a  daily  basis. 

Vendor  personnel  also  maintain  their  system  knowledge  by  working  with 
the  products  constantly. 


•  As  equipment  becomes  more  reliable  and  fewer  service  calls  are  made, 
detailed  knowledge  of  the  systems  is  lost  and  must  be  regained  each  time 
service  is  required. 

This  problem  is  aggravated  at  the  ammunition  plants  by  long,  dormant 
periods  that  may  exist  between  reactivations. 


Systems  maintenance  personnel  at  some  ammunition  plants  claimed 
that  being  away  from  the  system  for  six  months  required  them  to  make 
extensive  reviews  of  system  material  before  they  could  confidently 
effect  reactivation. 

•  Exhibit  Vll-5  conceptually  shows  the  effect  of  elapsed  time  between  calls  on 
learning  retention.  In  the  dormancy  situation,  with  long  periods  between 
service  calls,  the  effect  is  greatly  intensified.  In  other  words,  the  '"scallops" 
will  be  more  pronounced  as  the  average  time  between  service  calls  increases. 

•  In  order  to  minimize  this  effect,  one  or  more  of  the  new  diagnostic  and 
maintenance  procedures  must  be  employed.  These  include  systems  support 
centers,  remote  diagnostics,  user  self-maintenance,  and,  possibly,  depot 
maintenance. 
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EXHIBIT  VlI-5 


EFFECT  OF  ELAPSED  TIME  BETWEEN 
CALLS  ON  RETENTION  OF  LEARNING 


TRADITIONAL  EQUIPMENT 
WITH  FREQUENTLY  REIN- 
FORCED LEARNING 
(SHORT  TIME  BETWEEN  CALLS) 


NUMBER  OF  CALLS 

GRAPH  iS  CONCEPTUAL  TO  DRAMATIZE  THE  EFFORT  OF  LEARNING  EXTINCTION  AS 
ELAPSED  TIME  BETWEEN  CALLS  INCREASES. 


•  Training  that  is  wholly  dependent  upon  the  system  vendor  is  at  the  mercy  of 
the  vendor's  business  decisions  and,  therefore,  is  not  in  the  long-term  best 
interests  of  any  mobilization  effort. 

Vendor  independent  training  capabilities  can  be  achieved  through  the 
use  of  outside  training  companies  or  through  the  operating  contractors' 
training  facilities. 

Training  by  lecture  alone  is  not  the  best  method.  If  the  training 
material  is  presented  via  lecture  alone,  much  detail  can  be  lost  with 
each  succeeding  lecture. 

The  objective  training  approaches  are  best,  such  as  computer-based 
instruction  or  simulation. 

C.       PERSONNEL  TRACKING 

•  Even  if  personnel  have  been  away  from  the  system  for  extensive  periods  of 
time,  they  can  recall  system  details  faster  than  new  people  can  be  trained. 

•  A  personnel  tracking  system  would  provide  a  thread  to  everyone  who  has 
experience  on  a  particular  system. 

System  vendors  can  usually  track  people  with  particular  skills  as  long  as 
they  remain  employed  with  the  vendor. 

Operating  contractors  sometimes  maintain  information  concerning  the 
work  histories  of  their  employees  and  could  "find"  their  people  with 
EPCS  experience  as  long  as  they  remained  employees. 
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•  Most  of  the  respondents  felt  that  people  experienced  with  a  particular  EPC 
system  would  cooperate  with  an  attempt  to  keep  track  of  skilled  personnel  in 
time  of  mobilization. 

Such  personnel,  it  was  felt,  would  be  willing  to  complete  an  annual 
questionnaire  on  their  current  location  and  status. 

This  kind  of  a  skills  inventory  program  must  be  undertaken  by  a  group 
such  as  ARRCOM  that  has  purview  over  all  such  systems. 

Keeping  track  of  all  people  who  worked  on  a  particular  system  at  an 
ammunition  plant  would  be  of  major  interest.  Of  interest  also  are  all 
people  who  have  experience  on  a  particular  system  regardless  of  the 
application. 

D.       PROFESSIONAL  SERVICES  COMPANIES 

•  Professional  services  companies  that  provide  software  services,  consulting 
services,  system  design,  etc.  for  the  manufacturing  industries  are  listed  in 
Appendix  C. 

•  These  companies  maintain  skilled  personnel  in  the  various  areas  in  which  they 
specialize. 

An  analysis  of  these  companies  will  uncover  those  that  have  skills  in 
areas  germane  to  the  needs  of  the  ammunition  plants,  and  adroit 
courting  of  the  companies  will  result  in  a  relationship  that  will  prove 
mutually  beneficial  in  times  of  emergency. 

in  any  geographic  area,  these  companies  form  a  skill  pool  that  might  be 
mobilized  in  times  of  emergency.  Each  ammunition  plant  with  an  EPC 
system  should  be  aware  of  the  professional  services  companies  in  the 
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local  areas  and  their  particular  expertise.  Relationships  should  be  built 
so  that  in  emergencies,  work  can  be  performed  with  a  minimum  of  red 
tape. 

E.       THIRD-PARTY  MAINTENANCE  COMPANIES 

•  The  types  of  TPMs  have  been  described  previously. 

A  list  of  the  major  TPMs  is  given  in  Appendix  D. 

•  The  major  national  TPMs  of  either  type  would  consider  a  contract  to  maintain 
the  EPC  systems  at  the  ammunition  plants  and  would  maintain  personnel 
trained  in  troubleshooting  the  installed  systems. 

The  TPMs  would  consider  only  multiplant  contracts  and  would  also 
require  a  preliminary  study  contract  to  determine  what  needed  to  be 
accomplished  at  each  plant. 

•  Many  TPM  companies  will  stock  spares  and  also  have  reliable  short-run 
production  capabilities  to  make  spares  that  are  no  longer  available  from  the 
vendor. 

Digital  Equipment  Corporation's  field  service  organization  has  its  own 
manufacturing  facility  and  will  produce  spare  parts  for  DEC  systems 
that  are  no  longer  In  production.  DEC  field  service  will  only  maintain 
DEC  equipment. 

•  Some  TPM  companies  such  as  GE  and  RCA  would  be  interested  in  maintaining 
not  only  the  electronic  equipment  but  the  entire  automated  production  line. 

•  Third-party  maintenance  companies  are  a  valuable  resource  that  ARRCOM 
should  cultivate. 
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VIII  CONCLUSIONS 


•  INPUT  found  no  industry  guidelines  for  long-term  storage  of  electronic 
equipment.  Thus,  there  are  no  established  "rules"  directly  transferable  to  the 
ammunition  plant  situation. 

Vendors  exhibited  little  interest  in  the  problem.  Their  thrust  is  toward 
reliability  for  systems  that  operate  24  hours  a  day,  365  days  each  year. 
The  fallout  from  this  goal  will  ease  the  long-term  system  maintain- 
ability problem. 

Few  users  were  found  with  storage  requirements  for  electronic  systems. 
One  leasing  company  stored  IBM  360  systems  for  two  years  in  a  heated 
warehouse.  Each  cabinet  was  kept  in  a  large  sealed  plastic  bag.  A 
minimum  number  of  problems  occurred  upon  planned,  deliberate  reacti- 
vation. !n  short,  systems  can  be  stored. 

Sufficient  information  is  being  gathered,  however,  under  the 
ARRADCOM  program  so  that  explicit  recommendations  on  storage 
conditions,  reactivation  procedures,  and  further  testing,  can  be  made 
with  a  high  level  of  confidence. 

•  The  effects  of  the  strong  trend  in  EPC  system  design  toward  distributed 
decision  and  control  will  be  evidenced  throughout  the  production  line. 
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Less  redundant  central  site  hardware  will  be  necessary  because  the 
central  site  will  beconne  a  management  information  collection  point. 
Control  will  be  effected  in  the  field  with  less  expensive  hardware. 

The  less  expensive,  versatile,  programmable  field  controllers  will 
provide  a  type  of  standardization  that  will  in  turn  mean  fewer  spares  to 
stock  and  easy  replacement,  leading  to  reduced  MTTR  and  negligible 
MTSF. 

Systems  manufacturers  have  recognized  the  advantages  of  providing  families 
of  systems  so  that  as  needs  change,  the  system  can  grow  at  minimal  cost  and 
minimal  downtime. 

Compatible  systems  also  allow  for  the  use  of  the  same  software, 
eliminating  costly  program  conversions. 

Software  will  become  more  standardized  between  vendors  as  a  result  of 
the  increasing  use  of  the  DOD-backed  language  ADA  to  be  used  for 
real-time  and  process  control  systems. 

The  increasingly  dense  chips  that  are  being  manufactured  now  and  are 
on  the  drawing  boards  will  provide  economic  incentives  to  the  vendors 
to  minimize  the  number  of  different  chips  used  and  to  maintain  the 
compatibility  of  their  systems  for  longer  periods.  Withdrawal  of  vendor 
support  for  a  system  will  be  a  less  likely  event  because  the  family  life 
will  be  extended. 

The  ammunition  plant  "design  to  operation"  cycle  that  INPUT  has 
witnessed  is  so  drawn  out  that  half  a  system's  life  cycle  has  passed 
before  the  production  line  is  functioning. 
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A.       AMMUNITION  PLANTS 


•  U.S.  Army  ammunition  plants  are  significantly  different  from  commercial 
EPCS  controlled  plants. 

The  EPC  system  at  a  commercial  plant  is  designed  for  round-the-clock 
operation. 

Automated  process  or  production  lines  at  ammunition  plants  are  also 
designed  for  round-the-clock  operation.  However,  automated  ammuni- 
tion plants  should  be  designed  to  reflect  their  anticipated  life-cycle 
mode  which  includes: 

A  full  production  mode. 

A  partial  production  mode. 

A  dormant  period. 

Several  reactivation  cycles. 

Several  layaway  cycles. 

This  lifetime  approach  to  the  design  of  automated  production  lines  will 
result  in  the  use  of  a  different  set  of  evaluation  criteria  when  selecting 
all  elements  of  the  process  line. 

•  Overall  plant  layaway  responsibility  lies  with  the  operating  contractor's 
facilities  engineering  group,  and  the  EPC  system  is  only  a  small  part  of  the 
total  plant. 

The  attitudes  expressed  by  the  facilities  engineering  people  reflected 
the  importance  of  the  one  production  line  to  the  entire  plant. 
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Frequently  no  one  person  seenned  to  have  special  layaway  responsibility 
for  the  EPC  systenn.  It  was  one  of  many  items  of  concern  and  since 
little  was  known  about  the  EPCs'  layaway  requirements,  no  special 
attention  is  given  to  the  system. 

The  most  interested  operating  contractor  personnel,  as  far  as  EPCS 
layaway  was  concerned,  were  those  people  responsible  for  the  system 
and  funded  to  explore  the  problems  of  layaway.  Those  plants  that  had 
EPC  systems  under  construction  or  in  prove-out  were  less  interested 
than  those  whose  system  was  currently  dormant. 

•  From  the  limited  number  of  interviews  done  at  the  plants  and  perhaps  because 
primarily  operating  contractor  personnel  were  interviewed,  factory  automa- 
tion programs  seemed  to  lack  a  long-term  coordinated  planning  effort  and/or 
group. 

This  is  evidenced  by  several  factors: 

The  multiplicity  of  contractors  involved  in  each  individual 
system  design. 

The  diversity  of  systems  installed. 

The  lack  of  any  sort  of  standardization  that  might  alleviate  the 
two  major  long-term  problems  of  spare  parts  and  trained 
personnel. 

The  lack  of  information,  though  it  may  exist,  of  system  lifetime 
cost  considerations.  Only  initial  procurement,  construction  and 
installation  costs  seem  to  be  considered  at  the  contract  award 
time. 
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•  A  negative  undertone  on  the  part  of  the  operating  contractors  was  detected  by 
INPUT.  This  was  not  true  in  all  cases,  but  it  occurred  enough  times  to  cause 
concern. 

This  attitude  seenned  similar  to  one  found  in  the  commercial  data 
processing  world  when  the  system  was  imposed  upon  a  group  without 
the  group's  full  and/or  sufficient  participation  in  the  system  selection, 
design,  etc.  Under  such  conditions,  it  is  very  difficult  to  prove  the 
system  beneficial  to  the  overall  effort. 

•  INPUT  found  some,  but  little,  evidence  of  in  plant  reliability  programs  in  place 
and  none  that  provided  feedback  to  ARRCOM.  Fortunately,  the  ARRADCOM 
MMT  program  is  addressing  this  specific  problem. 

Failure  reporting  systems  varied  from  plant  to  plant.  Some  were  more 
organized  than  others,  few  accumulated  information  over  time  and 
organized  the  data  for  management,  and  even  fewer  claimed  that 
management  acted  on  the  data  presented  or  even  knew  of  its  existence. 
None  indicated  that  ARRCOM  would  ever  be  notified  in  any  formalized 
fashion  of  failure  trends.  This  is  being  addressed,  as  discussed  before, 
in  the  MMT  program. 

•  No  spare  parts  management  system  has  been  organized  for  these  EPC  systems, 
and  the  operating  contractors  have  few  guidelines  of  their  own. 

Spare  parts  have  usually  been  stored  in  a  reasonable  manner  for  the 
short  term,  but  this  is  because  it  is  easier  to  leave  them  in  their 
original  packaging,  and  it  is  more  convenient  to  keep  the  parts  near  the 
computer;  therefore  the  parts  are  in  a  controlled  environment.  This  has 
occurred  on  an  ad  hoc  basis  in  the  absence  of  any  set  of  proper 
procedures. 

Spares  do  not  have  the  status  in  the  Army's  procurement  scheme  that 
INPUT  feels  they  should  have  due  to  their  critical  role. 
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Spare  parts  requirements  are  not  considered  during  the  initial  procure- 
ment cycle  but  are  only  included  in  operating  funds. 


No  funds  have  been  allocated  for  the  testing  of  spares  or  for  test 
equipment  to  do  so. 

The  retention  of  trained  personnel,  the  training  of  new  employees,  and  the 
retraining  of  skilled  people  is  a  major  problem  for  the  operating  contractors 
and,  therefore,  for  the  U.S.  Army. 

Vendor-supplied  training  is  sufficient  initially,  but  this  training  is  lost 
when  the  system  is  dormant. 

The  reactivation  of  the  system  periodically  could  provide  a  reasonable 
training  activity.  However,  regulations  prohibit  the  use  of  reactivation 
funds  for  training  purposes. 

As  systems  become  more  reliable  and  as  the  military  munitions  industry 
practices  "good"  layaway  techniques,  the  reactivations  will  go  increas- 
ingly smoothly  and  there  will  be  less  and  less  opportunity  to  trouble- 
shoot  the  system  at  that  time. 

Operating  ammunition  plants  and  plants  that  have  EPC  systems  under 
construction,  in  prove-out,  etc.  do  keep  skilled  engineering  and  mainte- 
nance personnel.  Once  the  plant  or  EPCS  is  laid  away,  these  people 
drift  away  to  more  challenging  positions.  Their  skills  are  lost  to  the 
U.S.  Army.  There  is  no  formal  ARRCOM-operating  contractor  system 
in  place  which  tracks  these  individuals  from  year  to  year  and  place  to 
place. 

The  multiplicity  of  contractors  and  the  resulting  dilution  of  responsibility  at 
any  one  EPCS  site  has  frequently  led  to  a  breakdown  in  communications, 
typified  in  Exhibit  Vlll-I,  and  sometimes  the  delivery  of  products,  especially 
software,  that  were  not  suitable  to  be  used  as  produced. 
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COMMUNICATIONS  -  A  KEY  FACTOR 


THIS  NEW  SYSTEM 
IS  WORKING  SO  WELL  AT 
WEFIANX  THAT  ENGINEERING 
/S  BLOWING  UP  WITH 
ENTHUSIASM. 
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As  a  result,  the  design,  construction,  installation,  testing,  and  operating 
cycles  become  drawn  out  over  long  periods  of  time  (almost  eight  years 
in  some  cases)  leading  to  a  frustrated,  demoralized  operating  group. 

B.  VENDORS 

•  Vendors  understood  the  Army's  problem,  claimed  a  willingness  to  cooperate 

but  were  unwilling  to  second-source  in-house  designed  components  and  were 
cagey  about  releasing  documentation  on  in-house  designed  components  even 
after  they  become  obsolete. 

This  is  due  to  existing  patents  and  product  protection. 

•  The  increasing  use  of  in-nouse  designed  components  gives  the  vendor  greater 
control  over  the  system's  life  and,  according  to  the  vendors,  will  increase 
reliability  but  could  have  both  a  positive  and  negative  effect  on  the  Army's 
needs. 

When  the  vendor  passes  on  documentation,  it  will  be  more  accurate  and 
in  greater  depth  than  was  previously  available. 

Second  sources  for  components  will  be  more  difficult  to  find. 

•  Mil-spec  components  are  not  used  in  off-the-shelf  equipment  because  the 
vendors  must  remiain  price  competitive. 

Some  vendors  are  burning-In  their  components  to  the  mil-spec  standard. 

Less  reliable  components  such  as  plastic  encapsulated  chips  and  UV 
alterable  proms  are  being  either  phased  out  or  not  used  in  critical 
portions  of  the  systems. 
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•  Users  making  a  major  investment  in  an  automated  plant  or  in  a  portion  of  the 
plant  tend  to  protect  that  investment  by: 

Using  a  single  overall  contractor  in  the  design  and  installation  of  the 
plant. 

Fully  participatng  as  the  owner/operator  in  all  aspects  of  the  design 
through  operation  cycle. 

Paying  top  dollars  to  the  system  vendor  to  accept  full  responsibility  for 
the  maintenance  of  the  system.  This  includes  stocking  spares,  keeping 
trained  maintenance  personnel  available  at  all  times,  frequently  on- 
site,  and  usually  a  contractual  commitment  to  some  uptime  perform- 
ance measure. 

•  It  appears  to  INPUT  that  users  of  electronic  process  control  systems  have 
little  inclination  for  doing  maintenance  work  on  their  own  or  of  using  depots, 
etc. 

At  this  point,  users  pay  for  on-site  vendor  maintenance  and  they  get 
what  they  pay  for. 

The  process  control  industry  doesn't  seem  ready  to  follow  the  mainte- 
nance trends  of  the  computer  industry  in  general.  The  vendors  will 
force  the  users  into  accepting  some  of  the  trends  as  costs  increase  and 
Skilled  personnel  become  increasingly  rare. 

•  Users  showed  a  greater  interest  in  the  layaway  problem  than  the  vendors,  but 
the  users  had  very  little  actual  experience  in  this  area. 

Several  are  very  interested  in  this  study. 
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Just  as  vendors  recommend  leaving  the  system  "on"  in  an  idling  mode 
for  laci<  of  better  information,  users  also  have  left  systems  "on"  out  of 
fear  of  the  unknown. 

Two  plants  are  being  used  as  test  sites  under  the  MMT  program  with 
results  to  be  applied  to  other  Army  facilities. 

•  After  the  initial  burn-in  period,  users  found  system  uptime  in  the  over  98% 
range  and  in  some  cases  close  to  ??.?%, 

Preventive  maintenance  time  is  primarily  used  for  work  on  the  electro- 
mechanical portions  of  the  system. 

Those  users  who  stocked  their  own  spare  parts  followed  the  vendor's 
recommendation  for  their  particular  configuration  and  modified  this 
practice  based  upon  their  own  in-plant  experience. 

•  Training  is  not  a  major  problem  at  user  sites.  Vendors  provide  the  initial 
training  and  daily  interaction  with  the  system  provides  the  refresher  course. 

•  Users  generally  do  plan  to  replace  systems  when  they  become  obsolete  which 
requires  careful,  detailed  planning  and  excellent  communication  between 
vendor  and  user  personnel  at  all  levels. 
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IX  RECOMMENDATIONS 


IX  RECOMMENDATIONS 


A.  GENERAL 


•  As  discussed  in  Chapter  Vlil,  the  conclusions  section  of  this  report,  INPUT 
found  that  a  large  number  of  Army  departments,  contractors  and  subcontrac- 
tors are  involved  in  the  specification,  design,  installation,  and  operation  and 
maintenance  of  automated  production  lines.  Between  these  elements,  com- 
munication is  generally  weak,  creating  some  confusion  about  responsibilities 
and  commitments.  Further,  coordination  is  sometimes  poor  and  objectives  and 
priorities  are  ill-defined  with  the  result  that  contractors  do  not  always 
perform  in  the  best  interests  of  the  Army. 

•  Three  recommendations  which  would  address  these  problems  are: 

Minimize  the  number  of  Army  units  and  contractors  associated  with  the 
program. 

Increase  the  Army's  supervisory  role. 

Institute  long-term  planning  with  continuity  as  a  primary  objective. 

•  Some  specifics  of  these  three  recommendations  are  delineated  in  the  following 
paragraphs. 
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I.        MINIMIZE  THE  NUMBER  OF  UNITS  INVOLVED 

•  INPUT  is  fully  aware  of  the  DOD  procurement  practices  that  can  lead  to  a 
proliferation  of  entities  being  involved  in  any  given  program.  It  seems, 
however,  that  rapid  reactivation  of  a  munitions  production  line  is  the 
overriding  program  objective. 

•  The  stages  encountered  over  the  life  cycle  of  a  production  line  are: 

Specification. 
Design. 

Construction/installation. 
Prove-out/acceptance  testing. 
Operation. 
Maintenance. 

•  In  the  best  of  all  possible  worlds,  one  organization  would  have  complete 
responsibility  for  all  stages. 

•  Exhibit  IX- 1  shows  a  series  of  alternative  "responsibility  scenarios"  going  from 
best  to  worst  case,  as  evaluated  by  INPUT. 

In  the  best  case,  recognized  as  unlikely,  ARRCOM  could  run  the  whole 
show. 

In  the  worst  case,  a  different  organization  has  responsibility  for  each 
stage.  The  worst  case  scenario  closely  resembles  the  situation  for 
many  of  the  automated  production  lines  in  existence  now. 
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EXHIBIT  IX-1 


ARRCOM  AND  CONTRACTOR  RESPONSIBILITIES 
ALTERNATIVE  SCENARIOS 


STAGE 


BEST 


SCENARIOS 


WORST 


SPECIFICATION 


DESIGN 


CONSTRUCTION 


PROVE-OUT 


OPERATION 


MAINTENANCE 


ARRCOM 


ARRCOM 


ARRCOM 


ARRCOM 


ARRCOM 


ARRCOM 


ARRCOM 
CONTRACTOR  A 
CONTRACTOR  A 
ARRCOM 
CONTRACTOR  A 
CONTRACTOR  A 


ARRCOM 

CORPS  OF 
ENGINEERS 

CORPS  OF 
ENGINEERS 

ARRCOM 

CONTRACTOR  A 
CONTRACTOR  A 


ARRCOM 

CONTRACTOR  A 

CORPS  OF 
ENGINEERS 

CONTRACTOR  A 

CONTRACTOR  A 

CONTRACTORS 
(MANY) 


ARRCOM 

CONTRACTOR  A 

CORPS  OF 
ENGINEERS 

CONTRACTOR  B 

CONTRACTOR  C 

CONTRACTORS 

(MANY) 
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2.  INCREASE  THE  ARMY'S  SUPERVISORY  ROLE 

•  Assuming  that  the  Army  cannot  achieve  the  best  case  scenario,  INPUT 
believes  it  is  imperative  that  ARRCOM  continue  and  be  increasingly  active  in 
supervising  all  activities  at  each  stage. 

Constant  monitoring  and  supervision  can  result  in  long-term  continuity. 

The  longevity  in  grade  of  civil  service  personnel  employed  by  the  Army 
is  likely  to  be  better  than  that  of  contractor  or  military  personnel. 

Potential  problems  or  departures  from  objectives  can  be  spotted  early 
so  that  anticipatory  corrective  actions  can  be  taken  in  a  timely  manner. 

3.  INSTITUTE  LONG-TERM  PLANNING 

•  An  automated  production  line  needs  to  be  planned,  evaluated  and  considered  in 
the  context  of  its  requirements  over  the  lifetime  of  the  line  -  typically  20 
years. 

INPUT  perceives  that  the  implementation  of  a  line  is  seen  as  a  one-shot 
task  and  that  long-term  implications  are  minimized. 

The  MMT  4322  program  is  addressing  this  situation. 

•  When  seen  in  a  long-term  context,  those  parameters  that  are  subject  to  change 
over  a  20-year  period  will  be  identified.  Once  identified,  contingency  and 
action  plans  can  be  developed  that  define  what  should  happen  in  the  event  of  a 
change. 

•  Some  items  that  can  be  better  dealt  with  as  a  result  of  thorough  long-term 
planning  are: 
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Costs  -  For  example,  when  costs  are  projected  over  a  20-year  life 
cycle,  tradeoffs  between  initial  expense  and  operating  expense  can  be 
better  analyzed. 

Personnel  -  For  example,  training  requirements  can  be  better  defined  if 
the  frequency  of  personnel  shifts  is  anticipated. 

States  of  readiness  -  For  example,  readiness  requirements  may  change 
as  a  result  of  the  global  political  situation.  Planning  for  these 
fluctuations  may  dictate  different  approaches  to  construction,  opera- 
tion and  maintenance. 

New  technology  -  For  example,  should  a  production  line  be  designed  at 
the  outset  to  accommodate  new  technology  that  may  not  be  expected 
for  ten  years? 

•  In  summary,  INPUT  recommends  that  a  readiness  function  be  Identified  and 
centralized  within  ARRCOM  to  focus  on  all  new,  existing,  and  planned  EPC 
systems  at  the  ammunition  plants.  This  single  group  would  have  to  be  at  a 
very  high  management  level.  The  center  would: 

Establish  standards  for  the  design,  procurement,  installation,  operation, 
sparing,  layaway,  reactivation  and  replacement  of  EPC  systems. 

Develop  a  long-term  (20  years)  plan  for  the  acquisition  of  EPC  systems. 

Delegate  responsibility  to  the  operating  contractors  for  appropriate 
portions  of  the  total  effort. 

Assist  the  ammunition  plants  in  planning,  designing  and  procuring  EPC 
systems. 

Independently  review  and  evaluate  the  operating  contractor's  EPC 
systems  development  plans,  designs  and  projects. 


-  233  - 


INPUT 


Establish  a  documentation  center  as  assessed  in  MMT  4322,  retaining  all 
the  documentation  on  all  the  systems  and  keep  the  documents  updated. 

Establish  a  spare  parts  inventory  management  system  that  would  keep 
track  of  the  use  and  location  of  spares  and  issue  guidelines  for  their 
storage. 

Establish  a  personnel  management  tracking  system  to  maintain  contact 
for  at  least  a  five-year  period  with  people  skilled  in  the  use  and 
maintenance  of  the  installed  EPC  systems. 

Retain  responsibility  for  all  EPC  systems'  prove-out  and  acceptance. 

Be  staffed  with  computer-skilled  personnel,  as  well  as  factory  automa- 
tion and  process  control  experts. 

Create  a  vendor  liaison  office  to  deal  with  suppliers  on  an  ARRCOM- 
wide  basis,  thereby  achieving  some  leverage. 

Operate  a  testing  program,  such  as  MMT,  aimied  at  resolving  some  of 
the  layaway  issues  that  are  unique  to  the  military  and  for  which  there  is 
no  industrial  experience. 

Aim  at  developing  a  strong  bond  of  cooperation  between  ARRCOM, 
ARRADCOM,  and  the  operating  contractors  and  also  among  the  operat- 
ing contractors  themselves. 

Apply  the  guidelines  of  MMT  4322  which  addresses  these  problems. 
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B.       SYSTEM  PROCUREMENT  RECOMMENDATIONS 


1.  STANDARDIZE 

•  Automation  equipment  and  software  should  be  standardized  across  similar 
production  lines  and,  whenever  feasible,  even  across  dissimilar  lines. 

This  implies  that  the  entire  munitions  production  program  be  perceived 
as  a  single  program,  and  not  as  a  mixture  of  independent  entities. 

•  Standardizing  on  equipment  means: 

More  incentive  to  selected  vendors  to  perform. 
Fewer  spares  requirements  because  of  parts  interchangeability. 
Fewer  training  programs.  ' 
Easier  transfer  of  people. 

Simpler  reliability  improvement  programs  since  information  need  be 
gathered  on  fewer  pieces  of  equipment. 

•  Standardizing  on  software  means  that,  regardless  of  the  equipment,  the 
software  interfaces  all  appear  the  same  to  the  applications  programmers. 

ADA  should  be  used  if  DOD  formally  adopts  it. 

2.  SEEK  VENDORS  WHO  CAN  DEMONSTRATE  LONG-TERM  COMMITMENTS 

•  Only  vendors  of  equipment  and  software  who  can  demonstrate  a  long-term 
commitment  to  the  products  they  would  provide  for  the  munitions  production 
lines  should  be  considered  as  viable  bidders. 
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•  Vendors  should  be  required  to  present  evidence  of  this  commitment  in  their 
proposals.  Some  of  the  items  that  can  be  construed  as  evidence  are: 

Written  commitment  to  provide  spares  over  the  projected  lifetime  of 
the  product  or  designation  of  second  sources  who  can  make  such  a 
commitment. 

Written  commitment  not  to  drop  support  when  systems  are  upgraded  or 
replaced. 

Fail-safe  provisions  for  turning  over  proprietary  drawings,  bills  of 
materials,  etc.  in  the  event  support  is  withdrawn  at  some  future  time. 

Progressive  maintenance  policies,  including  programs  for  customer 
training  and  assistance  with  provision  for  distributing  these  programs  to 
munitions  plants  in  remote  locations  at  reasonable  costs. 

Superior  facilities  for  equipment  and/or  software  diagnosis  and  repair 
including  such  things  as  remote  diagnostics,  easily  accessible  repair  and 
spares  depots,  centralized  fault/fix  data  bases,  etc. 

3.       SPECIFY  REQUIREMENTS 

•  INPUT  believes  that  the  long-range  needs  of  the  munitions  production  lines  are 
not  always  translated  into  requirements  specifications  for  system  procure- 
ments at  the  component  or  subsystem  level.  In  other  words,  the  implications 
of  long  dormancy,  fast  reactivation,  etc.  over  a  20-year  span  are  not 
considered. 

In  future  procurements,  these  requirements  should  be  specified  at  a 
very  detailed  level,  and  subsystem  or  component  bidders  should  be 
required  to  address  these  specifications. 
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In  relation  to  the  twin  problems  of  reliability  and  maintenance,  the  items  that 
should  be  detailed  in  the  specifications  are: 

Documentation  -  current  and  upgrades. 

Support  facilities  -  type  and  location. 

Training  -  methods  and  frequency. 

Maintenance  procedures. 

Spares  -  stocking  levels. 

Layaway  -  specifications  and  procedures. 

Reactivation  procedures. 

Layaway  requirements  should  be  included  initially  in  every  system  procure- 
ment. Requests  should  be  made  to  vendors  to  provide  information  on  the 
effects  of  long-term  storage  on  the  equipment  proposed.  Pressure  to  force 
vendors  to  respond  can  be  accomplished  in  two  woys; 

Seek  the  cooperation  of  other  military  and/or  federal  agencies  who 
have  similar  requirements  in  jointly  making  known  that  the  vendors  who 
will  undertake  such  accelerated  life  testing  will  gain  rating  points  in 
any  competitive  bidding  situation. 

Make  this  a  part  of  the  scope  when  purchasing. 

Require  vendors  to  respond  to  a  section  in  each  EPC  system  bid 
solicitation  that  requests  information  on  the  ease  of  layaway  and  the 
effects  of  dormancy.  Make  it  known  that  vendors'  responses  will  be  an 
important  part  of  the  vendor  responsiveness  evaluation  scheme. 
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INPUT  recognizes  that  this  is  difficult  but  believes  that  it  is 
important. 

•  Each  EPCS  bid  solicitation  should  contain  a  section  on  lifetime  costs,  and  the 
estimated  lifetime  costs  should  be  made  a  significant  factor  in  choosing  a 
vendor  or  evaluating  the  responses  to  the  solicitation. 

Each  solicitation  should  require  the  vendors  to  estimate  the  lifetime 
costs  of  the  proposal  based  upon  a  series  of  different  possible  events 
that  might  occur,  such  as  those  shown  in  Exhibit  IX-2.  The  solicitation 
would  have  to  include  a  list  of  cost  assumptions  such  as  power,  labor, 
etc.  and  would  have  to  ask  the  vendor  to  supply  cost  breakdowns 
including  spare  parts,  preventive  maintenance,  etc.  If  several  vendors 
make  a  joint  bid,  then  each  should  respond  fully  for  the  subsystem  bid. 

C«       MAINTENANCE  PROCUREMENT  RECOMMENDATIONS 

I.  SPARES 

•  Access  to  cn  adequate  supply  of  spares  over  the  projected  system  lifetime 
must  be  ensured. 

Consideration  should  be  given  to  buying  spares  initially  or  at  least 
allocating  the  funding  for  future  spares  procurement  along  with  the 
initial  procurement  contract. 

Provision  of  funds  should  also  be  made  for  purchasing  test  equipment  to 
test  incoming  spare  parts  on  a  sample  basis. 

•  A  backup  program  (should  manufacturers  go  out  of  business  or  otherwise  drop 
support)  for  spares  replacement  must  be  instituted. 
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EXHIBIT  IX-2 


SUGGESTED  EPC  SYSTEM  USAGE  ALTERNATIVES 


YEARS 

ALTERNATIVES 

1 

2 

3 

4 

0  - 

ONE  SHIFT 

ONE  SHIFT 

ONE  SHIFT 

ONE  SHIFT 

FULL  PRODUCTION 

FULL  PRODUCTION 

FULL  PRODUCTION 

FULL  PRODUCTION 

LAYAWAY  AND 

LAYAWAY  AND 

LAYAWAY  PERIOD 

REACTIVATION 

REACTIVATION 

ON  SIX  MONTH 

ON  TWO  YEAR 

CYCLES 

CYCLES 

REACTIVATION 

AND  TRAINING 

5  — 

LAYAWAY  PERIOD 

DIN c  bnl r  1 

FULL  PRODUCTION 

THREE  SHIFTS 

FULL  PRODUCTION 

10  — 

ONE  SHIFT 

HALF  PRODUCTION 

15  — 

LAYAWAY  PERIOD 

REACTIVATION 

AND  TRAINING 

LAYAWAY  PERIOD 

20  — 

!  ^ 
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A  program  for  testing  spares  is  needed,  (At  one  user  facility,  INPUT  was  told 
that  50%  of  spares  were  found  to  be  defective.) 

Test  specifications  must  be  established.  In  most  cases,  the  manufac- 
turers' specifications  will  be  adequate. 

If  standardization  of  systems  can  be  achieved  over  a  broad  enough  base, 
ARRCOM  should  consider  purchasing  a  backup  system  that  could  be  moved 
into  any  site  to  replace  a  down  system  that  could  not  be  repaired  within 
reactivation  time  requirements. 

This  system  should  be  kept  in  a  controlled  environment  off-site  from 
any  munitions  plants. 

The  system  should  be  kept  in  top  working  order  at  all  times. 

Fault/fix  information  from  this  backup  system  should  also  feed  into  the 
centralized  data  base  as  part  of  the  reliability  improvement  program. 

Guidelines  in  MMT  4322  address  this  issue. 

PERSONNEL  TRAINING 

A  system  for  keeping  track  of  all  personnel  skilled  at  diagnosing  or  repairing 
equipment,  software  or  systems  should  be  established  after  they  have  retired 
or  left  the  employ  of  the  Army  or  contractors.  The  government  could  then 
attempt  to  call  on  these  people  in  case  of  national  emergency. 

Such  programs  exist  within  some  operating  contractor's  organizations 
and  within  some  vendor  organizations,  but  a  broader  program  uniformly 
administered  is  necessary. 
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3.  TRAINING  AND  RETRAINING 

•  Provision  should  be  made  for  an  ongoing  formal  training/retraining  program. 

Newer  techniques,  such  as  video  or  computer-based  instruction  should 
be  considered,  especially  for  retraining. 

•  Reactivation  periods  should  be  used  as  on-site  training  and  retraining  labora- 
tories. 

Army  regulations  should  be  modified  so  that  this  process  is  legal. 

4.  ESCALATION  PROCEDURES 

•  Suitable  escalation  procedures  for  both  prime  and  all  subcontractors  should  be 
clearly  stipulated  to  ensure  all  possible  resources  are  brought  to  bear  on 
difficult  maintenance  problems,  and  that  contractor/subcontractor  responsi- 
bilities are  clearly  delineated.  This  should  eliminate  the  common  "finger- 
pointing"  syndrome. 

The  optimum  repair  level  analysis  refers  to  this,  according  to 
ARRADCOM. 

5.  RELIABILITY  IMPROVEMENT  PROGRAM 

•  An  ongoing  reliability  improvement  program  should  be  established.  The 
program,  at  a  minimum,  should  consist  of: 

A  reporting  system  wherein  each  fault  and  fix  is  reported  centrally. 

A  procedure  for  analyzing  the  resulting  data. 

A  system)  to  provide  feedback  to  all  parties  who  can  effect  improve- 
ments. 
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•  ARRCOM  should  consider  using  the  facilities  of  one  of  the  computer  system 
vendors  since  most  are  well  set  up  to  handle  this  kind  of  program. 

If  this  is  not  feasible,  ARRCOM  should  establish  and  operate  its  own 
system.  The  Defect  and  Failure  Reporting  form,  discussed  earlier, 
should  address  this  issue. 

6.        CHOICE  OF  MAINTENANCE  PROVIDER 

•  "       INPUT'S  recommendation  that  ARRCOM-ARRADCOM  handle  its  own  mainte- 

nance notwithstanding,  there  ore  several  choices  of  maintenance  provider. 

The  plant  operating  contractor. 
The  equipment  manufacturer(s). 
A  third-party  maintenance  (TPM)  vendor. 

•  The  best  of  these  alternatives  is  the  plant  operating  contractor  because  a 
single  entity  can  be  held  accountable  for  the  functioning  of  the  line  or  plant. 

•  The  second  choice  is  the  process  control  system  equipment  provider,  since  it  is 
closest  to  the  equipment  and  can  presumably  draw  upon  superior  talent. 

Because  of  the  remote  geographic  distribution  of  the  equipment, 
vendors  may  not  be  interested  in  assuming  prime  responsibility. 

•  In  the  event  the  preferred  choices  are  not  viable,  TPM  vendors  should  be 
considered. 

INPUT  recommends  the  "facilities  TPM"  type  of  vendor. 

These  are  usually  divisions  of  very  large  companies  which 
presumably  would  offer  a  better  chance  for  long-term  viability 
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because  they  have  superior  financial,  personnel,  and  technical 
resources. 

It  is  doubtful  that  a  facilities  TPM  vendor  would  take  a  small  contract. 
Therefore,  ARRCOM  should  consider  awarding  large  contracts  that 
either  cover  nnultiple  plants  or  cover  more  equipment  than  just  the 
process  control  system  in  a  single  plant. 

An  example  of  the  latter  would  be  to  include  such  electronic 
systems  as  fire  control,  closed  circuit  TV,  telephones,  etc. 

•  In  any  case,  regardless  of  the  type  of  maintenance  vendor  selected,  contracts 
should  provide  strong  financial  performance  incentives  in  order  to  help  ensure 
that  performance  objectives  are  achieved. 

•  Contractor  personnel  interviewed  by  INPUT  generally  stated  that  they  saw 
automated  control  systems  as  such  a  small  piece  of  the  overall  plant  that  they 
gave  it  relatively  low  priority. 

ARRCOM  should  require  that  one  person  with  the  operating  contrac- 
tor's facilities  engineering  group  (or  another  group  that  has  layaway 
responsibility)  have  sole  responsibility  for  EPCS  layaway.  This  person 
should  have  adequate  funds  available  to  do  a  complete  job.  Some 
computer  orientation,  especially  a  maintenance  background,  would  be 
an  essential  requirement  for  this  position. 

7.       LINES  OF  COMMUNICATION 

•  The  maintenance  contract  should  clearly  stipulate  how  the  maintenance 
vendor  should  communicate  with  equipment  manufacturers,  other  contractors, 
and  the  Army. 
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In  particular,  documentation  flow  should  be  spelled  out  as  part  of  a  plan 
that  delineates  responsibilities,  reporting  systems,  and  lines  of 
authority. 


D.       LAYAWAY  AND  REACTIVATION  RECOMMENDATIONS 


I.       LAYAWAY  RECOMMENDATIONS 

•  INPUT  recommends  that  the  layaway/reactivation  cycle  be  extended  from  the 
successful  six-month  period  to  the  longest  period  found  among  the  inter- 
viewees -  two  years. 

This  two-year  cycle  should  be  considered  an  experiment  and  should  be 
properly  monitored.  At  least  a  six-year  test  should  be  considered. 

The  test  should  be  carried  out  in  three  parts: 


Equipment  should  be  stored  in  a  controlled  environment  with  a 
temperature  of  70*^  and  humidity  at  40%,  just  as  the  Joliet  AAP 
equipment  was  stored  during  the  three-year  storage  with  six- 
month  test  intervals. 

Equipment  should  be  stored  in  a  heated  area  and  an  appropriately 
sized  capsule  be  used  in  the  equipment  enclosure  as  an  anticor- 
rosion  agent.  The  cabinet  should  be  covered  with  ordinary 
plastic  to  keep  them  dust  free. 

Equipment  should  be  stored  in  a  heated  area  and  enclosed  in 
sealed  plastic  bags.  These  bags  should  be  ziplocked  if  possible 
rather  than  heat-sealed  because  heat-sealing  frequently  gives  off 
gases  which  could  harm  the  metallic  surfaces.   The  plastic  bags 
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should  be  tested  to  see  if  harmful  vapor  results  from  the  heat- 
sealing  process. 


Caution  -  ZERUST®  capsules,  which  may  be  used,  exude  a  vapor  that 
protectively  coats  the  metal  surfaces.  INPUT  has  seen  no  tests  that 
show  that  the  vapor  does  or  does  not  have  to  be  removed  before  the 
equipment,  using  the  extremely  low  current  levels  found  in  modern 
computers,  can  be  activated.  If  the  vapor  must  be  removed  before  the 
components  will  work  properly,  then  the  ZERUST®  products  should  not 
be  considered  for  the  application.  INPUT  urges  that  complete  tests  be 
carried  out  before  this  product  is  used  for  the  storage  of  modern 
electronic  equipment.  General  effectiveness  test  results  can  be  found 
in  Appendix  E. 

•  Valves,  actuators  and  sensors  can  remain  on  the  line  if  tests  show  that  they 
can  survive.  If  the  pipes,  vessels,  cabinets,  and/or  machinery  in  which  they 
function  can  be  closed  off,  then  capsules  or  bags  as  appropriate  should  be  used. 

For  long-term  storage,  the  best  procedure  is  to  disassemble  the  valves, 
actuators,  sensors,  etc.,  clean  all  parts  thoroughly,  and  store  in  sealed 
bags. 

•  Paper  products  such  as  documentation,  punched  cards  and  paper  tape  should  be 
stored  in  several  places  within  the  plants  and  at  the  technical  center  at 
ARRCOM-ARRADCOM. 

Paper  products  should  be  kept  in  a  more  humid  environment  than 
electronic  equipment. 

Spare  parts  should  not  be  stored  in  brown  (Kraft)  paper  bags  or  in 
corrugated  cardboard  bins.  Paper  products  are  made  with  a 
sodium  sulfate  solution,  and  the  sulphur  will  harm  the  parts  if 
they  are  stored  for  long  periods  of  time. 
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The  parts  should  be  stored  in  ziplocked  plastic  bags  or  in  ZERUST® 
plastic  bags  if  ZERUST®  passes  the  current  level  test. 

A  two-year  or  longer  dormant  period  seriously  affects  retention  of  system 
knowledge  by  the  operating  contractor's  skilled  people. 

INPUT  recommends  the  adoption  of  a  computer-based  instructional 
program  such  as  PLATO  by  Control  Data  Corporation  or  a  simulator 
training  program  that  will  simulate  a  "hands  on"  training  environment. 

REACTIVATION  RECOMMENDATIONS 

The  basic  procedure  when  reactivating  a  system  after  it  has  been  dormant  is 
to  very  carefully  follow  the  vendor's  procedure  for  startup  as  if  the  system  is 
brand  new. 

It  is  imperative,  therefore,  that  the  operating  contractor's  personnel  or 
whoever  will  have  reactivation  responsibility  be  present  at  the  initial 
startup,  document  the  procedure  thoroughly,  and  obtain  full  documenta- 
tion of  the  process  from  the  vendor.  This  documentation  must  be  kept 
up-to-date. 

The  MMT  4322  procedures  will  supplement  this  procedure  and  may  be 
used  during  start-up. 

The  packing,  whether  plastic  bags  or  something  else,  must  be  undone  carefully 
so  as  to  minimize  disturbing  the  dust  accumulation. 

If  not  covered  and/or  kept  in  a  controlled  environmient,  the  parts  must 
be  vacuumed  thoroughly  and  carefully,  with  special  attention  to 
electromechanical  units  such  as  disk  drives  and  printers. 
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•  A  thorough  visual  inspection  of  all  units,  cabinets,  subassemblies,  boards,  chips 
and  other  components  must  be  undertaken.  This  was  stressed  as  being  an 
especially  important  procedure  because  many  potential  problems  can  thus  be 
averted. 

Boards,  chips,  etc.  should  not  be  unseated  as  a  matter  of  course.  They 
should  be  reseated  if  they  appear  loose. 

Cabling  must  be  very  carefully  inspected  and  should  be  handled  to  test 
for  brittleness. 

Antioxidants  from  commonly  used  insulating  materials  evaporate 
over  time  (regardless  of  whether  the  equipment  is  in  storage  or 
not)  leaving  the  insulation  very  brittle  and  subject  to  cracking. 
If  this  occurs  the  cables  must  be  replaced. 

Some  insulating  materials  are  less  likely  to  become  brittle. 
These  should  be  specified  in  all  contracts  calling  for  long-term 
use  of  the  cables,  especially  if  dormant  periods  are  anticipated. 

The  mechanical  subassemblies  of  electromechanical  equipment  must  be 
examined  for  grease  deterioration.  All  equipment  on  the  entire 
production  line  that  requires  grease  for  proper  operation  must  be 
similarly  inspected. 

Studies  should  be  undertaken  to  find  or  develop  greases  less 
subject  to  deterioration,  and  they  should  be  specified  in  EPC 
system  procurements. 

•  if  the  storage  period  is  greater  than  two  years,  all  gaskets,  0-rings,  seals,  etc. 
should  be  discarded  and  replaced  with  new  ones. 

These  are  usually  standard  items  and  should  be  purchased  fresh  for  each 
reactivation. 
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Recalibration  must  also  be  done  on  all  devices  that  have  been  disassem- 
bled and  require  calibration. 

•  All  equipment  should  be  brought  up  slowly  in  20°^  increments,  keeping  it  a  few 
hours  at  each  level. 

The  equipment  is  maintained  under  "no  load"  conditions  for  24  hours. 

Special  attention  must  be  paid  to  the  slow  reforming  of  the  capacitors 
and  to  all  power  supplies,  as  done  successfully  at  JAAP  and  VAAP. 

All  special  cautions  and  other  instructions  from  the  vendors  should  be 
followed  unless  experience  shows  that  they  may  not  be  necessary. 
However,  certain  warranties  might  be  voided  by  not  following  the 
vendor's  instructions  carefully. 

•  All   loops  should  be  tested  and  brought  up  individually  before  testing  for 
interaction  between  loops. 

•  A  precise  failure  reporting  system  should  be  in  place  before  the  reactivation 
starts,  as  done  under  the  ARRADCOM  MMT  4322  program. 

Not  only  must  the  fault  be  well  documented,  but  the  symptom  and 
troubleshooting  procedures  must  also  be  thoroughly  described  as  part  of 
a  good  management  feedback  system. 

E.       EXISTING  FACILITIES 

•  Existing  facilities  should  be  brought  up  to  the  level  of  new  procurements  if  all 
the  recommendations  of  this  report  are  followed. 
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A  person  within  each  operating  contractor's  organization  should  be 
assigned  responsibility  for  layaway,  and  that  function  should  be 
adequately  funded. 

A  spare  parts  managennent  system  should  be  instituted  with  strict 
adherence  to  dating,  storing  and  testing  procedures. 

A  documentation  hierarchy  should  be  developed,  and  the  documents 
collected  and  organized  along  with  thorough  procedures  for  updating. 

Vendor-independent  training  programs  should  be  developed. 

•  During  construction  and  installation  of  the  system,  since  the  cycle  is  so  drawn 
out,  new  equipment  should  be  treated  as  if  it  were  in  layaway. 

•  The  MMT  4322  reportedly  addresses  a  number  of  these  needs. 
F.       PCP-88  SYSTEMS  RECOMMENDATIONS 

•  The  following  recommendations  pertain  to  the  PCP-88  systems  at  JAAP, 
VAAP  and  RAAP  only.  They  have  been  singled  out  because  the  vendor  of 
these  systems,  the  Foxboro  Company,  inc.,  has  announced  the  withdrawal  of 
support  as  of  December  31,  1982. 

•  Withdrawal  of  support  by  Foxboro  means  that: 

Spare  parts  will  no  longer  be  produced  and,  as  existing  supplies  are 
bought  up,  spares  will  become  increasingly  rare.  Existing  PCP-88 
systems  may  serve  as  a  source  of  spare  parts. 
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Documentation  will  not  be  updated  and  as  supplies  run  out  will  not  be 
available.  Hopefully,  large  system  changes  will  not  be  made  and 
existing  documentation  will  be  useful. 

Formal  training  courses  may  not  be  given  and  training  material  will  not 
be  available. 

Maintenance  services  will  not  be  guaranteed  on  a  formal  basis  from 
Foxboro. 

The  ammunition  plants  will  be  largely  "on  their  own"  either  singly  or 
jointly  with  respect  to  the  automated  lines  using  the  PCP-88  control 
system. 

•  All  the  preceding  recommendations  apply  to  these  systems.  The  following 
recommendations,  while  similar  to  some  of  the  preceding,  should  be  instituted 
more  rapidly  and  more  thoroughly  than  may  occur  with  new  systems. 

•  INPUT  recommends  that  the  PCP-88  systems  be  kept  and  maintained  as  long 
as  possible  and  that  everything  necessary  be  done  to  foster  this  effort. 

The  PCP-88  system  is  a  proven  system  with  uptimes  in  commercial 
experience  of  greater  than  98%. 

The  PCP-88  system  has  successfully  survived  layaway  and  reactivation 
periods  and  has  shown  no  indication  of  not  being  able  to  continue 
further  with  such  a  program. 

The  spare  parts,  personnel,  documentation  problems,  etc.  caused  by 
Foxboro's  dropping  of  support  for  the  PCP-88  can  be  greatly  alleviated 
by  prompt  action  on  the  programs  outlined  in  this  report. 

•  INPUT  recommends  that  a  single  office  or  group  at  ARRCOM-ARRADCOM  be 
created  and  charged  with  the  full  responsibility  of  implementing  all  programs 
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necessary  to  maintain  the  three  PCP-88  systenns.  This  must  be  done  whether 
or  not  a  systems  development  center  is  established  at  ARRCOM-ARRADCOM. 
These  programs  should  include: 


A  documentation  program  with  all  hardware  and  software  documents 
stored  in  a  central  location  under  conditions  designed  to  best  preserve 
and  safeguard  them. 

This  documentation  must  be  obtained  from  Foxboro  and  all 
subcontractors  as  necessary.  Proprietary  documentation  should 
be  releasable  once  the  product  is  no  longer  manufactured. 

Detailed  vendor  lists  are  a  necessary  part  of  the  program. 

A  spare  parts  program  that  would  maintain  a  complete  set  of  spares  and 
emphasize  those  spares  that  experience  difficulties  or  high  failure  rates 
and  which  studies  show  to  be  most  vulnerable. 

Testing,  dating,  cycling,  repairing  and  proper  storage  are  basic 
elements  of  the  program. 

Also  this  program  must  locate  and  maintain  a  list  of  companies 
that  will  duplicate  any  component  in  short-run  quantities. 

Maintenance  of  an  up-to-date  second  source  file  is  a  major 
activity  of  this  office. 

A  vendor-independent  training  capability  must  be  established  and 
maintained. 

Elements  of  this  capability  exist  currently.  This  should  be 
expanded  and  updated  if  necessary.  Training  materials  should  be 
purchased  from  Foxboro  and  all  other  pertinent  subcontractors. 


-  251  - 


INPUT 


Additional  programs  as  deemed  necessary  to  maximize  the  life  of  the 
systems  and,  therefore,  the  Army's  return  on  its  investment. 

An  important  program  must  be  undertaken  to  determine  if  the 
PCP-88  system  can  effectively  control  multiple  TNT  lines  simul- 
taneously. The  software  must  be  debugged  and  the  effect  of  the 
multi-line  interaction  on  production  capability  must  be  esti- 
mated. 

The  operating  contractors  must  be  represented  in  this  group. 

INPUT  recommends  that  large  national  third-party  maintenance  companies  be 
offered  the  opportunity  to  maintain  the  PCP-88  system. 

These  companies  should  be  asked  to  guarantee: 

To  keep  a  cadre  of  trained  personnel. 

To  procure  spare  parts  as  needed  through  purchase  or  manu- 
facture. 

To  reactivate  the  system  periodically  as  determined  by  the 
appropriate  Army  commands. 

INPUT  suggests  that  if  the  U.S.  Army  is  going  to  proceed  with  its  moderniza- 
tion program  at  the  ammunition  plants  in  general,  and  at  VAAP  and  JAAP  in 
particular,  then  the  Army  should: 

Automate  the  other  lines  using  the  latest  hardware,  software  and 
process  knowledge. 

When  these  lines  are  proved-out  then  replicate  these  new  systems  on 
the  PCP-88  lines. 
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Adopt  systems  that  have  been  proved-out  on  other  CIL  continuous 
process  lines  to  avoid  the  costly  and  disruptive  conversion  process. 

Consider  separately  the  systenn  at  RAAP,  since  at  this  time  it  is  a 
single  system  for  a  unique  process.  , 

•  if  the  U.S.  Army  chooses  not  to  automate  additional  CIL  continuous  process 
lines,  then  at  some  point  conversion  will  be  required.  ARRCOM-ARRADCOM 
should  consider  adopting  the  Navy's  project  management  and  control  system 
for  use  in  estimating  and  managing  such  conversion. 
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APPENDIX  B:  DEFINITIONS 


•  AUTQMATbD  REMOTE  DIAGNOSTICS  -  A  remote  diagnostic  system  that 
interfaces  with  an  automated  analysis  routine  at  the  remote  assistance  center. 

•  BRING-UP  DIAGNOSTICS  -  Specialized  diagnostics  designed  during  the  equip- 
ment development  phase  to  verify  the  functional  integrity  of  various  stages  of 
design.  Usually  incorporated  into  the  field  diagnostic  package  as  verification 
routines. 

•  CLOSED  LOOP  (feedback  loop)  -  A  signal  path  which  includes  a  forward  path, 
a  feedback  path  and  a  summing  point,  and  forms  a  closed  circuit. 

•  COMPUTING  INSTRUMENT  -  A  device  in  which  the  output  is  related  to  the 
input  or  inputs  by  a  mathematical  function  such  as  addition,  averaging, 
division,  integration,  lead/lag,  signal  limiting,  squaring,  square  root  extraction, 
subtraction,  etc. 

•  CONTROL  ROOM  -  A  location  with  heat  and/or  cooling  facilities.  Conditions 
are  maintained  within  specified  limits.  Provisions  for  automatically  maintain- 
ing constant  temperature  and  humidity  may  or  may  not  be  provided.  Note: 
Control  room  areas  are  commonly  provided  for  operation  of  those  parts  of  a 
control  system  for  which  operator  surveillance  on  a  continuing  basis  is 
required. 
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CONTROL  SYSTEM  -  A  system  in  which  deliberate  guidance  or  manipulation 
is  used  to  achieve  a  prescribed  value  of  a  variable. 


•  CONTROL  VALVE  -  A  final  controlling  element  through  which  a  fluid  passes, 
which  adjusts  the  size  of  flow  passage  as  directed  by  a  signal  from  a  controller 
to  modify  the  rate  of  flow  of  the  fluid. 

•  CONTROLLER  -  A  device  which  operates  automatically  to  regulate  a  con- 
trolled variable. 

•  DIAGNOSTIC  AIDS  -  Various  devices  and  resources  that  assist  in  the  effort  to 
diagnose  equipment  problems.  Some  examples  are:  diagnostics,  remote 
diagnostics,  error  logs  (manual),  on-board  statistical  error  logs,  documenta- 
tion, special  test  equipment,  FE  panels,  etc. 

•  DIAGNOSTICS  -  Programmed  routines  used  to  verify  the  functional  integrity 
of  computer  equipment,  to  force  failures,  report  on  failure  patterns,  and  to 
isolate  faults  through  fault-locating  techniques. 

•  DISTRIBUTED  PROCESSING  -  Tne  deployment  of  programimable  intelligence 
to  the  site  where  the  particular  processing  function  is  performed.  Computers, 
controllers  and  devices  are  interconnected  through  communications  lines 
adapted  to  Individual  user  needs. 

•  DISTRIBUTOR  -  Purchases  the  programmable  controllers,  sensors,  actuators, 
etc.  on  an  OEM  basis  from  the  manufacturer  and  markets  them  to  the  end 
user. 

•  END  USER  -  May  buy  a  system  from  the  hardware  supplier(s)  and  do  his  own 
programming,  interfacing,  and  installation.  Alternatively,  may  buy  a  turnkey 
system  from  a  systems  house  or  hardware  integrator. 
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ENGINEERING  CHANGE  NOTICE  -  Notice  of  improvements  or  corrections  in 
a  product  after  it  has  been  released  to  production  or  has  been  installed  at  the 
user's  site. 

ENGINEERING  CHANGE  ORDER  (ECO)  -  Instructions  including  bill  of 
material  and  parts  required  to  effect  the  engineering  change. 

FLTs  -  A  specialized  diagnostic  package  designed  to  isolate  fault  in  a  failing 
computer  device.  The  abbreviation  of  "Fault-Locating  Tests." 

FIELD  ENGINEER  (FE)  -  An  individual  who  responds  to  a  user's  call  for  service 
and  repairs  a  device  or  system.  FE  is  used  interchangeably  with  customer 
engineer,  serviceperson,  maintenance  person,  etc. 

FLOWMETER  -  A  device  that  measures  the  rate  of  flow  or  quantity  of  a 
moving  fluid  in  an  open  or  closed  conduit,  it  usually  consists  of  both  a  primary 
and  a  secondary  device.  Note:  It  is  acceptable  in  practice  to  further  identify 
the  flowmeter  by  its  applied  theory;  as  differential  pressure,  velocity,  area, 
force,  etc.  or  by  Its  applied  technology  as  orifice,  turbine,  vortex,  ultrasonic, 
etc.  Examples  include  turbine  flowmeter,  magnetic  flowmeter,  fluidic 
pressure  flowmeter,  etc. 

HARDWARE  -  Physical  equipment  directly  involved  in  performing  Industrial 
process  measuring  and  controlling  functions. 

INSTRUMENTATION  -  A  collection  of  Instruments  or  their  application  for  the 
purpose  of  observation,  measurement  or  control. 

MEAN  TIME  BETWEEN  FAILURES  (MT3F)  -  The  average  of  a  series  of 
measured  elapsed  time  intervals  between  reported  failures  on  a  device  or 
system. 
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MEAN  TIME  TO  REPAIR  -  The  average  of  a  series  of  measured  elapsed  time 
intervals  between  a  field  engineer's  arrival  at  the  user's  site  and  the  repaired 
device's  return  to  full  operation. 

MEAN  TIME  TO  RESPOND  -  The  average  of  a  series  of  measured  elapsed  time 
intervals  between  a  user's  service  call  and  a  field  engineer's  arrival  at  the 
user's  location. 

MEAN  TIME  TO  SYSTEM  FAILURE  -  The  average  of  a  series  of  measured 
elapsea  time  intervals  between  instances  of  total  system  failure. 

MODULE  -  An  assembly  of  interconnected  components  which  constitutes  an 
identifiable  device,  instrum.ent,  or  piece  of  equipment.  A  module  can  be 
disconnected,  removed  as  a  unit,  and  replaced  with  a  spare.  It  has  definable 
performance  characteristics  which  permit  it  to  be  tested  as  a  unit.  Note;  A 
module  could  be  a  card  or  other  subassembly  of  a  larger  device,  provided  it 
meets  the  requirements  of  this  definition. 

OPEN  LOOP  -  A  signal  path  without  feedback. 

PERIPHERALS  -  Induce  all  input,  output,  and  storage  devices,  other  than 
main  memory,  which  are  locally  connected  to  the  main  processor  and  are  not 
generally  included  in  other  categories,  such  as  terminals,  sensors,  actuators, 
etc. 

-PROCESS  CONTROL  -  The  regulation  or  manipulation  of  variables  influencing 
the  conauct  of  a  process  in  sucn  a  vay  as  to  obtain  a  product  of  desired  quality 
and  quantity  in  an  efficient  manner. 

PROCESS  MEASUREMENT  -  The  acquisition  of  information  that  establishes 
the  magnitude  of  process  quantities. 

PROGRAMMABLE  CONTROLLER  -  A  controller  which  automatically  holds  or 
changes  a  set  point  to  follow  a  prescribed  program  for  a  process. 
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•  .    QUALITY  ASSURANCE  DIAGNOSTICS  -  Combinations  of  manual  tests  and 

algorithms  run  under  variable  conditions  of  environmental  stress  to  ensure  that 
equipment  runs  properly  at  and  beyond  specified  condition  limits.  Seldom  used 
by  field  engineers  and  only  recommended  in  extreme  cases  within  the  user's 
environment. 

•  RELIABILITY  -  The  probability  that  a  device  or  system  will  perform  its 
objective  adequately,  for  the  period  of  time  specified,  under  the  operating 
conditions  specified. 

•  REMOTE  DIAGNOSTICS  (RD)  -  Diagnostics  run  by  the  vendor  from  a  remote 
location  without  the  intervention  of  the  user's  operator;  diagnostics  run  by  an 
on-site  field  engineer  tied  to  a  central  support  center,  or  by  a  user  tied  to  a 
central  support  center.  It  can  usually  isolate  a  fault  to  the  lowest  exchange- 
able unit. 

•  SET  POINT  -  An  input  variable  which  sets  the  desired  value  of  the  controlled 
variable.  Note  I ;  The  input  variable  may  be  manually  set,  automatically  set, 
or  programmed.  Note  2;  It  is  expressed  in  the  same  units  as  the  controlled 
variable. 

•  SHARED  TIME  CONTROL  -  Control  in  which  one  controller  divides  its 
computation  or  control  time  among  several  control  loops  rather  than  by  acting 
on  all  loops  simultaneously. 

•  SUPERVISORY  CONTROL  -  Control  in  which  the  control  loops  operate 
independently  subject  to  intermittent  corrective  action;  e.g.,  set  point  changes 
from  an  external  source. 

•  SYSTEM  SUPPORT  CENTER  (SSC)  -  A  central  technical  support  facility 
staffed  by  highly  skilled  field  engineers  and  accessed  over  a  national  "hotline" 
number.  A  system  support  center  is  available  to  both  users  and  field  engineers 
for  the  analysis  of  problems  in  hardware,  software  or  a  combination  of  the 
two. 
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USER  SELF-MAINTENANCE  (USM)  -  Some  involvement  by  individual  users 
the  installation,  diagnosis  and  repair  of  their  own  installed  equipment. 

WARM-UP  PERIOD  -  The  time  required  after  energizing  a  device  before 
rated  performance  characteristics  apply. 
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APPENDIX  C:         COMPUTER  PROFESSIONAL  SERVICES  COMPANIES 


A.       LIST  I 


•  Revenues:  More  than  $5,000,000. 

•  Over  50%  of  revenue  obtained  from  software  products  and  professional  services. 

•  Over  30%  of  revenue  obtained  from  the  manufacturing  sector  of  the  economy. 

Allen  Services  Corporation 
212  W.  National  Road 
Vandalia,  OH  45377 
(513)  890-1200 
A.  Allen,  President 

Applied  Information  Development  (AID),  Inc. 

823  Commerce  Drive 

Oak  Brook,  IL  60521 

(312)  654-3030 

Lee  Munder,  President 

Cincom  Systems,  inc. 
2300  Montana  Avenue 
Cincinnati,  OH  4521  I 
(513)  662-2300 
Thomas  M.  Mies,  President 


-  27!  - 


INPUT 


I 

Cogna  Systems  Corporation 
222  North  Dearborn 
Chicago,  IL  60601 
(312)  431-6700 
J.M.  Roe,  President 

Cullinane  Database  Systems,  Inc. 

400  Blue  Hill  Drive 

Westwood,  MA  02090 

(617)329-7700 

John  J.  Cullinane,  President 

Information  Science,  Inc. 
95  Chestnut  Ridge 
Montvale,  NJ  07645 
(201)  391-1600 
Jack  Might,  President 

Informatics,  Inc. 
21031  Ventura  Blvd. 
Woodland  Hills,  CA  91364 
(213)  887-9040 

Walter  F.  Bauer,  Ph.D.,  President 

Logicon,  Inc. 

21535  Hawthorne  Blvd. 

Torrance,  CA  90503 

(212)  540-9200 

John  R.  Woodhull,  President 

Pansophic  Systems,  Inc. 

709  Enterprise  Drive 

Oak  Brook,  IL  60521 

(312)  986-6000 

Joseph  A.  Piscopa,  President 
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Scientific  Software  Corporation 
633  17th  Street 
Denver,  CO  80202  ' 
(303)  571-111! 

E.A.  Breitenbacin,  Ph.D.,  President 

Software  Ag  of  North  America,  Inc. 

1  1800  Sunrise  Valley  Drive 
Reston,  VA  22091 

(703)  860-5050 

John  Maguire,  President 

Software  International  Corporation 

2  Elm  Square 
Andover,  MA  01810 
(617)  475-5040 

William  G.  Watson,  President 

Structural  Dynamics 
Research  Corporation 
2000  Eastman  Drive 
Milford,  OH  45156 
(513)  576-2400 
Jason  R.  Lemon,  President 

LIST  il 

Revenues:  More  than  $5,000,000. 

Over  50%  of  revenue  obtained  from  software  products  and  professional  services. 
Some  revenue  from  the  manufacturing  sector. 
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AGS  Computers,  Inc. 

I  I  35  Spruce  Drive 
Mountainside,  NJ  07092 
(201)  654-4321 

Lawrence  J.  Schoenberg,  President 

Advanced  Computer  Techniques  Corporation 
437  Madison  Avenue 
New  York,  NY  10022 
(212)  421-4688 

Charles  P.  Lecht,  President/Chairman 

Analysts  International  Corporation 
7615  Metro  Blvd. 
Minneapolis,  MN  55435 
(612)  835-2330 

Frederick  W.  Lang,  President 

Auxton  Computer  Enterprises 

1345  Avenue  of  the  Americas 

New  York,  NY  10019 

(212)  489-7620 

John  P.  Croxton,  President 

C.A.C.I.,  Inc. 

181  5  Ft.  Meyer  Drive 
Arlington,  VA  22209 
(703)  841-7800 

William  W.  Fair,  Ph.D.,  President 

CGA  Computer  Associates,  Inc. 
25  Commerce  Drive 
Cranford,  NJ  07016 
(201)  272-3900 

Bernard  M.  Goldsmith  111,  President 
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Computer  Company  (The) 
1905  Westmoreland  Street 
Richmond,  VA  23230 
(804)  358-2171 

Walter  R.  T.  Witschey,  President 

Computeristics,  Inc. 
Benson  Road 
Middlebury,  CT  06749 
(203)  573-2222 

James  H.  Chambers,  President 

Computer  Sciences  Corporation 
650  North  Sepulveda  Blvd. 
El  Segundo,  CA  90245 
(213)615-0311 

William  R.  Hoover,  President 

Computer  Task  Group,  Inc. 
800  Delaware  Avenue 
Buffalo,  NY  14209 
(716)  882-8000 

David  N.  Campbell,  President 

Comtech  Group  International  Ltd. 
5  Fairview  Mall  Drive 
Suite  380 

Willowdale,  Ontario  NM2J2 

(416)  492-1480 

Donald  McPhail,  President 

Consumer  Systems  Services  Group,  Inc. 

2107  Swift  Drive 

Oakbrook,  IL  60521 

(312)  325-2109 

Ray  M.  Dionisio,  President 
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Cutler  Williams,  Inc. 
2655  Villa  Creek  Drive  ' 
Dallas,  TX  75234 
(214)  243-3421 
George  Enochs,  President 

Cybernetics  and  Systems,  Inc. 

6100  Dutchman's  Lane 

Louisville,  KY  40205 

(502)  459-8080 

Al  Bailey,  President 

DASD  Corporation 
9045  N.  Deerwood  Drive 
Suite  301 

Milwaukee,  Wl  53209 
(414)  355-3405 

Martin  R.  Marshall,  President 

Data  Communications  Corporation 
3000  Directors  Row 
Memphis,  TN  38131 
(901)  345-3544 
President,  Chairman 

Caller  Associates,  Inc.  (GAI) 
1815  North  Fort  Myer  Drive 
Arlington,  VA  22209 
(703)  525-9292 
Jerry  Caller,  President 

Intel  Systems  Corporaticn 

12575  Research  Blvd. 

P.O.  Box  9968 

Austin,  TX  78766 

(512)  258-5171 

Kent  Ochei,  President 
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Keane,  Inc. 

210  Commercial  Street 

Boston,  MA  02109 

(617)742-5210 

John  F.  Keane,  President 

Litton-Mellonics 

1 00 1  West  Maude  Avenue 

Sunnyvale,  CA  94086 

(408)  245-0795 

Richard  W.  Baker,  President 

Martin  Marietta  Data  Systems 
300  East  Joppa  Road 
Baltimore,  MD  21204 
(301)  321-5700 

Richard  J.  Walters,  President 

Monchik-Weber  Corporation 

I  I  I  John  Street 

New  York,  NY  10038 

(212)  962-2400 

Stewart  Monchik,  President 

Payne  Consultants,  Inc. 
Time  &  Life  Building 
Rockefeller  Center 
New  York,  NY  10022 
(212)  581-5500 
Bruce  Payne,  President 

Project  Software  &  Development,  inc. 

I  4  Story  Street 

Cambridge,  MA  02138 

(617)  661-1444 

Robert  L.  Daniels,  President 
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Rand  Information  Systems,  Inc. 

98  Battery  Street 

San  Francisco,  CA  941  I  I 

(415)  392-2500 

E.  Donald  Shapiro,  Chairman 

Security  Mutual  Casualty  Data  Services 
222  South  Riverside  Plaza 
Chicago,  IL  60606 
(312)  621-4612 

Elwood  D.  Yeater,  Vice  President 

Spiridellis  &  Associates,  Inc. 
133  Avenue  of  the  Americas 
New  York,  NY  10036 
(212)  221-7270 

Nikolaous  T.  Spiridellis,  President 

Systems  and  Programming  Resources,  Inc. 
1211  W.  22nd  Street 
Suite  600 

Oakbrook,  IL  60521 
(312)  654-4188 

Systems  Architects,  Inc. 
Thomas  Patton  Drive 
Randolph,  MA  02368 
(617)  961-^^840 
George  Pan,  President 

Systems  Engineering  Computer  Co. 

P.O.  Box  26729 

Richmond,  VA  23260 

(804)  281-6456 

John  F.  Noll,  President 
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Telstat  Systems,  Inc. 
60  Hudson  Street 
New  York,  NY  10013 

(212)  227-5082 

Bruce  Hechler,  President 

Telos  Computing,  Inc. 
3231  Ocean  Park  Blvd. 
Santa  Monica,  CA  90405 

(213)  450-2424 

Len  Conger,  President 
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APPENDIX    E:    ZERUST    VAPOR    CAPSULE  EVALUATION 


SORTHERN  INSTRUMENTS  CORPORATION 


SWITCHGEAR  BOXES  -   VAPOR  CAPSULES  EVALUATION 

The  attached  describes  the  results  of  a  series  of  tests 
on  ZERUST  Vapor  Capsules  carried  out  in  Japan.     In  this 
test  capsules  were  placed  in  electrical   switchgear  boxes 
and  the  boxes  were  subjected  to  the  following  conditions: 

-saltspray 

-  temperature  and  humidity  cycling 

-  constant  heat  and  humidity 

-  corros  i  ve  gases 

also  observed  was  the  effect  of  ZERUST  vapors  on  solder 
and  relay  contacts. 

As  you  will  note,  the  test  results  were  quite  favorable. 


NORTHERN   INSTRUMENTS  CORPORATION 


Vincent  J.  Graziano 


Name 


S  i  g  n^t  u  r  e 

President 


Title 


Date 


May  31  ,   1 979 
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ZERUST  VAPOR  CAPSULE  EVALUATION   -  JAPAN,  1978 


A.     Overall   Objective  and  Scope  of  Evaluation 

To  ascertain  the  corrosion  inhibiting  effectiveness  of 
ZERUST  Vapor  Capsules  in  accelerated  laboratory  tests 
for  the  purpose  of  utilizing  them  in  equipment  situated 
in  factory  areas  with  high  humidity  and  corrosion  atmos- 
pheres. 


B..     Specific  Objective  and  Tests 

1 .  Obj  ect  i  ve 

To  determine  the  corrosion  resistance  of  ZERUST  Vapor 
Capsules  in  a  seaside  location. 

Test  Method 

A  3  cycle  test  was  performed  where  enclosures  with 
test  specimens*  were  subjected  to  a  5%  salt  spray 
for  6  hours  and  left  standing  for  16  hours  (MIL-STD- 
1 01 D  for  72  hours  )  . 

2 .  Objective 

To  determine  the  corrosion  resistance  and  stability 
of  ZERUST  Vapor  Capsules  in  a  tropical  environment. 

Test  Method 

Test  specimens  were  subjected  to  the  following  cycling: 
25°C-    65<^C-    25°C-    55°C-    25°C-    -10°-    25  =  C 
in  a  relative  humidity  of  90-98?^  ( MI  L-STD-1 06D  for  10 
cycl es ) . 

3 .  Obj  ect  i  ve 

To  determine  the  corrosion  resistance  of  ZERUST  Vapor 
Capsules  in  an  environment  of  constant  heat  and  humidity 

Test  Method 

Test  Specimens  were  subject  to  40°C  t  2°C  and  90-95% 
relative  humidity  for  240  hours   (MIL-STD-1 03B ) . 
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4 .  Objective 

To  determine  the  effect  of  ZERUST  vapors  on  solder. 
Test  Method 

The  tin,  gold,  copper  and  specimens  from  test  1   -  3 
above  were  tested  regularly  for  soldering  wetness 
using  a  Meniscograph. 

5 .  Objective 

To  determine  the  corrosion  resistance  of  h  gram  COR 
TABs   in  a  corrosive  environment. 

Test  Method 


Test  specimens  were  subjected  to  fumes  of  H^S,  NO  , 
HCl   and  NH^.  ^ 

Objective 


To  determine  the  effect  of  ZERUST  Vapor  Capsules  on 
the  contact  resistance  of  relays. 

Test  Method 

The  contact  resistance  of  a  typical   relay,  24  VDC, 
protected  oy  ZERUST  Vapor  Capsules  was  checked  after 
30,000,   100,000,   300,000  and  500,000  make/break 
operations  and  compared  to  an  unprotected  relay. 

The  relay  (Diagram  2)  was  tested  in  the  circuit  shown 
in  Diagram  1.     The  circuit  cycled  9  times  per  second. 


Test  Resul ts 

1.  In  the  5o  salt  spray  test  there  was  marked  corrosion 
on  the  unprotected  iron,  aluminum,  brass  and  copper 
specimens.     However  on  the  ZERUST  protected  samples, 
the  iron  and  aluminum  were  only  slightly  rusted,  while 
brass  and  copper  suffered  no  corrosion,  not  even  dis- 
coloring. 

2.  In  the  temperature  and  humidity  cycle  test  there  were 
no  characteristic  changes   in  the  protected  specimens. 
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3.  In  the  constant  heat  and  humidity  test,  while  the 
ZERUST  protected   iron  and  copper  specimens  were  not 
completely  unchanged  as  they  were  in  test  2  above, 
there  was  a  great  difference  between  the  unprotected 
specimens  and  the  ZERUST  protected  specimens.     As  a 
result,   the  ZERUST  Vapor  Capsules  were  deemed  par- 
ticularly effective  in  such  conditions. 

4.  The  ZERUST  vapors  had  no  effect  on  soldering  applied 
tothe  test  specimens. 

5.  The  corrosive  environment  tested  showed  that  the 
ZERUST  vapors  would  effectively  neutralize  H2S  vapors 
on  a  long  term  basis  up  to  a  concentration  of  0.5  ppm. 
Short  term  protection  could  be  expected  if  the  con- 
centration rose  to  1.0  ppm.     In  a  typical   control  room 
of  a  steel   plant  ov  refinery  H^S  concentrations  have 
been  measured  at  0.05  ppm. 

6.  The  relay  test  showed  that  there  was  absolutely  no 
change  in  contact  resistance  of  a  protected  relay. 


*  Test  Specimens  - 


Fe  - 

•  iron 

Al  - 

-  aluminum 

Bs  - 

■  brass 

Zn  - 

■  zinc 

Ni  - 

■  nickel 

Cr  - 

■  chromium 

Sn  ■ 

-  tin 

Ag  - 

-  silver 

Au  - 

■  gold 

Cu  - 

-  copper 
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Diagram  #1 

=.£1-lA 
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Switch  gear  box 
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NORTHERN  INSTRUMENTS  CORPORATION 


PACKAGING,   SMALL  ARMS   -   HUMIDITY   EXPOSURE  TRIALS 


The  following  pages  list  the  results  of  a  series  of  tests 
wherein  various  preservative  systems  were  evaluated  for 
protecting  small   arms  against  humidity. 

The  tests  were  performed  by  the  Canadian  Department  of 
National   Defense.     The  humidity  cycle  to  which  all  samples 
were  subjected  was  that  of  MIL-STD-810C,  method  5021,  pro- 
cedure V.     This  procedure  called  for  480  hours  of  temperature 
cycling  between  23°C  and  54°C  in  90  to  95%  Relative  Humidity. 

The  results  are  reported  in  terms  of 

-  control  number  assigned  to  weapon 
-serial  numberofweapon 

-  type  of  wrapping  into  which  weapon  was  placed 

-  Type  of  corrosion  inhibitor  product  (preservative) 
tested 

-  condition  of  weapon  at  conclusion  of  test 

-  what  preparation  was  required  to  issue  the  weapon 
for  use 


The  tests  show  the  excellent  results  obtained  using  ZERUST 
Vapor  Capsules  (identified  in  these  tests  as  NIC-1  and  NIC-2) 
in  these  severe  test  conditions. 
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Commen  t s : 

1.  It  was  evident  from  the  condition  of  the  card- 
board boxes  and  the  swollen  and  warped  woodwork 
that  the  test  conditions  were  very  severe.  Any 
of  the  weapons  that  survived  the  test  would  cer- 
tainly last  for  an  extended  period  of  time  under 
normal   storage  conditions. 

2.  The  poly  type  barrier  material  has  the  added  ad- 
vantage of  being  able  to  read  the  serial  numbers 
without  disturbing  the  preservation. 
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NORTHERN  INSTRUMENTS  CORPORATION 


INTEGRATED  CIRCUITS  AND  ELECTRONIC  COMPONENTS 
-VAPOR  CAPSULE  EVALUATION- 


Humidity  and  a  corrosive  environment  have  adverse  effects 
on  integrated  circuits,  circuit  board  soldering  and  various 
electronic  components.     The  attached  describes  the  detailed 
results  of  an  extensive  evaluation  of  the  anti-corrosion 
effectiveness  of  ZERUST  vapors.     The  tests  were  performed 
in  Japan  by  the  Taiyo  Petroleum  Gas  Co.  Ltd. 

We  are  pleased  to  report  that  the  tests  showed  that  ZERUST 
Vapor  Capsules  performed  excellently  in  all  respects.  The 
vapors,   while  providing  the  requisite  protection  against 
corrosion,   did  not  have  any  side  effects  on  the  non-electrical 
materials  normally  found  v/ith  integrated  circuits  such  as 
connector  insulation,   epoxy,  plastic  housing,  etc. 

NORTxHEP^^   INSTRUMENTS  CORPOPJ^TION 


VINCENT  J.  GRAZIANO 


Title 

August  16,  1979 


Date 
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ZERUST  VAPOR  CAPSULE  EVALUATION 


Test  Conditions 


Tap  water  was  placed  in  a  large  glass  container.  Two 
ZERUST  VCI-1  Vapor  Capsules  were  then  suspended  in  the 
container  near  its  top.     The  capsule  vapors  were  allowed 
to  saturate  the  interior  of  the  container  for  a  48  hour 
period  before  the  test  components  were  introduced  into 
it.     The  test  components  were  then  placed  on  a  wire  mesh 
suspended  above  the  water  line.     The  test  tem.perature 
was^ maintained  between  102°F    (25°C)    and  10 8°F^  ( 2 8 °C ) . 
Relative  humidity  varied  between  95%  and  98%. 

The  status  of  the  integrated  circuits  was  checked  after 
2200  hours.     The  remaining  components  were  checked  after 
4000  hours,   the  results  of  which  are  listed  in  the  fol- 
lowing pages. 
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ZERUST  VAPOR  CAPSULE  EVALUATION 
Page  #2 


MATERIAL 
!•      I.e.    SN7405    (TTL  TI) 


Plastic  Container 

(silver  plate) 
Performance 


2 .  Printed  Circuit  Board 

Soldering  Pattern 

Connector  Tubing 

(gold  plate) 
Epoxy  (FR-4) 

Soldering  Resistance 
Soldering 

3 .  Connector 

a)  M  Connector  (AxM?) 

Terminal    (gold  plate) 
Flousing    (phenol  resin) 

Metal  Latch  Fittings 
Printed  Name 
Resistance 

b)  CIS  Connector  (AMPl 

Terminal  (foil) 
Housing  (plastic) 

Resistance 

c)  Flat  Cable  Connector 
3413-0 00 OT  (Sumitomo-3M) 

Terminal  (tin) 
Housing  (plastic) 

Resistance 

d)  Flat  Cable  Connector 
65483-023 


Terminal  (gold  plate) 
Housing  (plastic) 

Resistance 


TEST  RESL'LTS 


normal 

normal,   no  corrosion 
A.C.   test  -  as  specified 
D.C.   test  -  as  specified 
(100  pieces) 


normal ,   no  corrosion 
soldering  normal 
normal 

normal ,   no  warping  or 
discoloration 

normal 

satisfactory 


normal 

normal,   no  deform.ation  or 
discoloration 

normal 

printina  normal 

4M  ohm  -  as  specified 


normal 

normal,   no  deformiation  or 

discoloration 
5M  ohm  -  as  specified 


normal 

normal,   no  deformation  or 

discoloration 
4M  ohm  -  as  specified 


normal 

normal,   no  deformation  or 

discoloration 
6M  ohm  -  as  specified 


-  308  - 


ZERUST  VAPOR  CAPSULE  EVALUATION 
Page  #3 


MATERIAL 


TEST  RESULTS 


4 .  Terminal 


a)     TBS  Type 


Terminal  (tin) 
Housing  (Diallyl 

Phthalate) 
Resistance 


b)     Y  Tvpe 


normal 
normal , 

6M  ohm  - 


no  deformation  or 
discoloration 
as  specified 


Terminal  (tin) 


normal 


5.     Wire  and  Cable 


a)      Flat  Cables 

Soft  Vinyl  Cover 
(tin  foil  wire) 


normal,  no  discoloration  or 

deterioration 
soldering  -  satisfactory 
insulation       >   1000  M  ohms 


b)     Ribbon  Cable 

PVC  Covered  Wire 
(tin  foil  wire) 


normal,   no  discoloration  or 

deterioration 
soldering  -  satisfactory 

insulation       >   1000  iM  ohms 


c)     Ylax  Wire  (overbridge 
polyethylene  wire) 

0.3,  0.5  mm-^  (tin  foil) 
0.5  mm^  twisted  pair 


normal,   no  discoloration  or 

deterioration 
soldering  -  satisfactory 
insulation     >   1000  M  ohms 


d)     Overbridge  Polyethylene 
Single  Core  

Nylon  Wire 

(tin  foil  wire) 


normal,  no 
soldering  ■ 
insulation 


discoloration 
■  satisfactory 
>   10  0  0  M  ohms 


e)      3  Wire  Cord  (Kawasaki 

N.   CSA  Type  

PVC  Coated 
AC  (copper) 


normal,  no  discoloration  or 

deterioration 
soldering  -  satisfactory 

insulation     >  1000  M  ohms 
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ZERUST  VAPOR  CAPSULE  EVALUATION 
Page  #4 


MATERIAL 


TEST  RESULTS 


f )      2  Wire  Sealed  Cord 
PVC  and  Polyethylene 
coated 

(tin  foil  wire) 


normal,   no  discoloration  or 

deterioration 
soldering  -  satisfactory 
insulation     >   1000  M  ohms 


g)     Choke  Transform.er 

Wiring  

Coated  Copper  Wiring 
with  soldered  tips 


normal,   no  discoloration  or 

deterioration 
solderincT  -  satisfactory 
insulation     >  1000  iM  ohms 


6.     Miscellaneous  Electronic 
    Ccm.ponents 


a )     Choke  Transformer  (TDK) 
Nylon  Bobbin 
Copper  Alloy  Fixturing 


b)      Relay  (Matsushita 

Den-ko)  

NF4E-24V  AE   13  5-4  4 
Plastic  Sealed  Relay 


c )     Switch  (Nikkai) 

SP-2  012  type  6A ,  12  5VAC 
with  2  plastic  keys 


d )      Condenser  (Matsuo) 

Metallized  Polyester 
Coatina 

FNX-HO,    2  2K,    2  0  0V 


e )     Condenser  (Matsuo) 
Wet  Tantalum 
200  uf  2  5V 
Tin  Plated  Leads 


normal,   no  discoloration  or 

deterioration 
soldering  -  satisfactory 
inductance  -  as  specified 


normal,   no  discoloration  or 

deterioration 
movement  -  normal 
terminal  solderina-satis factory 
contact  resistance  25M  ohms 


normal,   no  discoloration 

or  deformation 
on/off  resistance  -  satisfactor 
terminal  solder ing-satisfactory. 


normal,   no  discoloration  or 

deformation 
soldering  -  satisfactory 
capacitance  -  as  specified 

normal,   no  discoloration  or 

deformation 
soldering  -  satisfactory 
caDacitance  -  as  specified 
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ZERUST  VAPOR  CAPSULE  EVALUATION 
Page  #5 


f-lATERIAL 


TEST  RESULTS 


f )  Resistor 
150K  ohm 


g)      Switch  (Digitran) 


Plastic  Housing 


h)  Bobbins 


With  Polyester 
Coated  Leads 


Lead  Insulation 
Contains  Silk 


i )     Power  Transistor 
2SD~4T7 


7.     Miscellaneous  Electrical 
 Supplies  

a)  Teflon  Conduit 

b)  Vinyl  Conduit  (clear) 

c)  Coil  Plug  (nylon) 

d)  Rubber  Door  Gasket 
(with  adhesives) 

e)  Acrylic  Circuit  Board 
(clear  and  green) 

f )  Vinylchloride  Base 

g)  Circuit  Board  Guide 
(Polvester  Carbonate ) 


normal ^   no  discoloration  or 
deformation 

solderina  -  satisfactory 
resistance  -  as  specified 


normal,   no  discoloration  or 

deformation 
movement  and  resistance  - 
satisfactory 


normal,   no  discoloration  or 
deformation 

soldering  -  satisfactory 
resistance  -  500  ohms  as 
specified 

normal,   no  discoloration  or 
deformation 

soldering  -  satisfactory 
resistance  -  500  ohms  as 
specified 


normal,   no  discoloration  or 
deformation 

soldering  -  satisfactory 


normal,   no  discoloration  or 

deformation 
normal,   no  discoloration  or 

deformation 
normal,   no  discoloration  or 

deterioration 
normal,   no  discoloration  or 

deterioration 

normal,  no  discoloration  or 
deterioration 

normal,   no  discoloration  or 
deterioration 

normal,   no  discoloration  or 
deterioration 
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NORTHERN  INSTRUMENTS  CORPORATION 


Laboratory  and  In-Field  Evaluation 

of  Vapor  Capsules*  -  Kansai   Electric  Power 

Company,  Japan 


The  attached  describes  the  results  and  provides  photographic 
documentation  of  evaluations  of  the  protection  afforded  by 
Northern  Instruments  vapor  capsules.     Two  series  of  tests 
were  conducted,  the  first  were  performed  in  the  LABORATORY, 
the  second  series  in  actual   FIELD  conditions.     Hence  the 
importance  of  these  results.     The  test  data  and  accompanying 
photographs  were  provided  by  Kansai  Electric  Power  Company 
of  Japan  . 

We  are  pleased  to  provide  you  with  the  impressive  results  of 
these  comparative  tests. 


NORTHERN   INSTRUMENTS  CORPORATION 


Vincent  J.  Graziano 


 President 

Title 


August  13,  1979 

Date 


*  In  Japan  ZERUST  Vapor  Capsules  are  distributed  under  the 
brand  name  "STOR-SAFE". 
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ZERUST  VAPOR  CAPSULES  EVALUATION  -   KANSAI   ELECTRIC  POWER  CO. 


Overall  Conclusion 

Vapor  phase  corrosion  inhibitors  have  been  manufactured  and 
widely  used  in  Japan  since  1958.     To  date,  the  most  well 
known  of  these  inhibitors   is  Dicyclohexylammonium  nitrite,  often 
referred  to  by  its  briefer  designator,  DICHAN.     This  product 
is  known  to  be  effective  on  ferrous  and  ferrous  alloys,  but  it 
attacks  many  non-ferrous  metals  and  additionally  requires  as 
neutral   an  environment  as  possible  when  used.     However  as  it  is 
a  recognized  inhibitor  it  was  decided  to  include  it  in  these  tests 
and  compare  the  effectiveness  of  STOR-SAFE  vapor  capsules  to  that 
of  DICHAN. 

After  months  of  testing,  ZERUST  Vapor  Capsules  were  shown  to 
outperform  DICHAN  in  every  test.     As  a  result  Kansai  Electric 
Power  stated  to  us  that  "STOR-SAFE  is  a  very  effective  vapor 
phase  ant i corros i ve  agent  for  protecting  industrial  meters  and 
similar  electrical   equipment  made  of  ferrous  and/or  non-ferrous 
materials  under  various  types  of  weather  conditions. 


I 
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ZERUST   VAPOR  CAPSULES   EVALUATION   -   KANSAI    ELECTRIC   POWER  CO. 


Laboratory  Tests 

Laboratory  tests  were  performed  under  conditions  designed  to 
accelerate  the  development  of  corrosion.     These  tests  were  con- 
ducted at  the  Kansai   Electric  Research  Laboratory  and  the  Taiyo- 
Ekka  Gas  Company. 

The  test  materials  were  as  shown  below: 

Material  Size  J I S  Gauge  Quantity 

Mild  Steel         30mm  x  2mm  JISG3101SS41  6  pes. 

Copper  30mm  x  2mm  JIS  H  3250  99.96':  6  pes. 


In  the  tests   two  specimens  were  protected  by  a  ZERUST  VC  1-1  cap- 
sule,  two  specimens  were  protected  by  an  amount  of  DICHAN  equivalent 
to  the  quantity  of  the   inhibitor  in  a  ZERUST  VC  1-1   capsule,  and  two 
specimens  were  not  protected. 

The  test  specimens  were  sealed  in  containers  with  500cc  of  distilled 
water  and  50ppm  of  H2S  gas  and  subjected  to  temperature  cycling  from 
15°C   (59°F)   and   50^C   (122°F)   for  one  month. 

Photographs  of  the  specimens  were  taken  at  the  conclusion  of  the  test 
and  are  shown  in  Exhibit  1  below. 


Exhibit  1 


STEEL 


COPPER 


NOT 
PROTECTED 


w 


ZERUST 
PROTECTED 


k 

V 


DICHAN 
PROTECTED 
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ZERUST  VAPOR  CAPSULES   EVALUATION   -   KA.NSAI   ELECTRIC  POWER 


F_i  e  1  d  Tests 

Field  tests  v;ere  performed  in  power  stations  situated  close 
to  a  petrochemical   refinery,   a  seaport  and  a  chlorination 
facility.     Steel   and  Copper  specimens  were  placed   in  pairs 
of  electrical   switching  boxes.     In  each  case  one  of  the  boxes 
was  protected  by  a  ZERUST  VC  2-1   capsule  and  the  other  was 
1  eft  unprotected  . 

The  results  of  these  tests  verified  those  obtained  in  the 
laboratory. 
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APPENDIX    F:    GAO    PROVISIONAL  CHECKLIST 

FOR  SOFTWARE 
DEVELOPMENT  CONTRACTING 


APPENDIX  F: 


GAO  PROVISIONAL  CHECKLIST  FOR  SOFTWARE 
DEVELOPMENT  CONTRACTING 


•  This  checklist  was  obtained  from  GAO  Report  rSMSD-80-4,  11/9/79,  Con- 
tracting for  Computer  Software  Development. 

•  This  checklist  covers  items  that  management  should  consider  when  contracting 
for  computer  software  development.  The  checklist  is  divided  into  five 
contract  phases:  pre-contract,  contracting,  proposal  evaluation,  performance 
period,  and  post-contract. 

•  While  this  checklist  is  only  an  interim  document,  it  will  be  useful  to  personnel 
involved  in  contracting  for  software  development. 

•  Some  of  the  points  included  in  this  list  might  apply  to  contracting  for  any  type 
of  service  and  may  be  covered  in  general  procurement  guidance.  Those  points 
are  listed  here  to  make  the  list  complete  and  to  emiphasize  that  they  can 
indeed  be  applied  to  software  development  contracts  as  well  as  to  others. 

A.       PHASE:  PRE-CONTRACT 

1.  *  OBJECTIVE 

•  To  obtain  realistic  assessments  of  the  size,  scope,  and  cost  of  the  effort  to 
produce  the  needed  software. 
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INPUT 


a.  Methods 


Describe  the  needed  software  as  completely  as  possible  in  the  request  for 
proposal  so  that  the  contractor  can  understand  and  address  the  full  scope  of 
work  in  his  proposal. 

Describe  the  software  so  that  people  not  familiar  with  agency  operations  can 
understand  the  need. 

Develop  criteria  to  evaluate  how  reasonable  and  accurate  the  contractor's 
costs  are. 

b.  Procedures 

Include  all  details  from  the  system  development  steps  completed  by  the 
agency  to  date. 

In  the  areas  where  detailed  specifications  have  not  been  developed,  clearly 
state  the  functional  requirements  the  software  must  satisfy. 

Give  all  known  constraints  and  parameters  the  vendor  must  work  with  in 
developing  the  software. 

c.  Possible  Ways  of  Accomplishing  the  Procedure 

The  use  of  agency  jargon,  which  might  be  unclear  to  outsiders,  should  be 
avoided. 

A  contractor's  accounting  system  must  be  determined  to  be  adequate  to 
generate  valid  cost  estimates. 

The  cost  of  past  development  efforts  may  be  compared  to  current  proposed 
costs. 
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INPL 


Caution  should  be  exercised  on  bids  that  are  much  higher  or  lower  than 
the  average  of  bids  received. 


2.  OBJECTIVE 

•  To  insure  the  contractor  nnost  suited  to  do  the  work  is  selected. 
0.  Method 

•  Develop  comprehensive  proposal  evaluation  standards  to  use  in  identifying 
nonresponsive  bidders  and  in  selecting  a  qualified  contractor. 

b.  Procedures 

•  Consider  the  contractor's  management  qualifications. 

Does  the  contractor's  organization  reflect  adequate  management 
overall? 

Is  adequate  management  available  for  this  project? 

Will  the  staff  responsible  for  the  proposal  also  work  on  the  project? 

Is  the  responsibility  for  this  project  at  a  high  enough  level  of  authority 
in  the  contractor's  organization? 

Is  it  likely  that  key  personnel  will  remain  with  the  project  from  start  to 
finish? 

•  Consider  the  contractor's  technical  approach. 

Does  it  reflect  a  knowledge  of  the  agency's  mission  and  functions? 
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INPUT 


Will  the  proposed  software  take  full  advantage  of  the  agency's  latest 
computer  hardware  capability? 

Does  the  contractor  use  software  performance  measurement  tools  and 
techniques  as  well  as  software  optimization  tools  to  insure  the  most 
efficient  development  possible? 

Does  the  technical  proposal  rely  on  state  of  the  art  or  unproven 
methodology  as  opposed  to  proven  technology? 

Is  the  overall  design  sound  and  feasible? 

•  Consider  the  contractor's  quality  assurance  program. 

Does  the  contractor  have  a  documented  program  in  effect  that  will 
ensure  that  the  software  meets  contract  specifications? 

Is  contractor  quality  assurance  measurement  compatible  with  the 
agency  acceptance  criteria  for  the  final  product? 

•  Consider  the  contractor's  cost  proposal. 

In  doing  so,  refer  to  the  criteria  in  the  first  objective  of  this  phase. 


B.       PHASE:  CONTRACTING 


OBJECTIVE 


•  To  initiate  a  framework  for  the  software  development  that  ensures  the 
maximum  chance  for  success  by  requiring  that  sound  system  development 
practices  be  followed. 
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INPL 


a.  Method 


Include  in  the  contract  a  requirement  that  the  contractor  not  bypass  system 
development  steps  in  the  development  process.  This  will  provide  a  means  for 
the  agency  to  assure  itself  that  the  contractor  is  performing  the  required 
system  development  steps. 

b.  Procedures 

Determine  the  system  development  steps  accomplished  by  the  agency  to  date. 

Identify  the  remaining  system  development  steps  to  be  accomplished  by  the 
contractor. 

Specify  the  sequence  in  which  these  steps  are  to  be  accomplished. 

c.  Possible  Way  of  Accomplishing  the  Procedures 

As  part  of  the  deliverables  that  the  contractor  must  provide,  include  a  product 
at  the  end  of  each  system  development  step  which  demonstrates  that  the  step 
has  been  satisfactorily  completed.  These  products  would  include  preliminary 
surveys,  feasibility  studies,  general  and  detail  design,  test  data  and  test  plan, 
the  actual  programs,  and  acceptance  test  results. 

OBJECTIVE 

Write  a  contract  which  clearly  states  the  obligations  of  both  the  agency  and 
the  contractor. 

a.  Methods 

Specify  in  the  contract's  statement  of  work  who  has  the  responsibility  for  each 
task. 
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INPU^ 


specify  in  the  contract  what  constitutes  satisfactory  performance  by  the 
contractor. 

b.  Procedures 

All  agency  obligations  should  be  included  in  the  milestone  chart  and  their 
performance  period  should  be  specified  the  same  as  the  contractor's. 

Tasks  should  be  scheduled  so  that  agency  work  that  is  prerequisite  to 
contractor  performance  will  not  delay  the  contractor. 

The  contract  should  provide  for  phasing  the  software  from  the  contractor  to 
the  agency,  including  agency  training  and  briefing  the  staff  during  the  work. 

c.  Possible  Ways  of  Accomplishing  the  Procedure 

Whether  the  specifics  of  the  software  are  known  or  not,  satisfactory  perform- 
ance should  be  quantified  in  terms  of  all  known  requirements  and  constraints. 
Some  areas  for  performance  standards  are  the: 

Ability  to  meet  agency's  functional  requirements. 

Growth  potential  requirements  of  the  system. 

Time  constraints  for  deliverables. 

Test  and  acceptance  criteria  which  must  be  met. 

Programming  language  standards  and  practices  to  be  followed. 

Documentation  standards  to  be  followed. 

Ease  of  modification. 

Maximum  computer  resources  allowed. 
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3.  OBJECTIVE 

•  Write  a  contract  which  gives  the  agency  the  right  and  the  visibility  to  manage 
the  contract. 

a.  Methods 

•  Provide  for  a  nnininnunn  of  investment  of  funds  before  the  quality  of  the 
contractor's  work  is  known. 

•  Specify  who  is  authorized  to  make  changes  in  the  contract. 

•  Provide  some  means  to  monitor  the  contractor's  progress. 

b.  Procedures 

•  Phase  the  development  into  logical  work  steps. 

•  Require  agency  approval  for  each  phase  before  beginning  work  on  the  next. 

•  The  more  undefined  the  software  is,  the  closer  the  phasing  steps  should  be  at 
the  outset. 

c.  Possible  Ways  of  Accomplishing  the  Procedures 

•  Besides  the  contracting  officer,  specify  management's  technical  representa- 
tive who  is  authorized  to  make  changes  and  to  answer  contractor's  questions. 

•  Use  phasing  to  assist  in  setting  up  a  milestone  chart  showing  the  timeframe 
for  each  work  step. 

•  Require  the  contractor  to  document   the  satisfactory  completion  of  each 
workstep  by  the  agreed  date. 


-  323  - 


INPUT 


•  Interim  progress  reports  may  be  required  from  the  contractor  to  allow 
progress  to  be  monitored  during  each  work  step. 

•  Identify  performance  as  well  as  functional  specifications. 

•  Specify  the  measures  of  reliability  and  quality  by  which  the  contractor's  work 
will  be  evaluated. 

4.  OBJECTIVE 

•  Write  a  contract  which  gives  the  agency  some  recourse  if  the  contractor  fails 
to  perform. 

a.  Methods 

•  The  contract  should  provide  for  the  agency  to  terminate  the  contract  if  it 
becomes  evident  that  the  contractor  cannot  perform  according  to  the  con- 
tract's terms. 

•  The  contract  should  also  provide  separate  due  dates  and  costs  for  each 
deliverable  and  a  provision  to  withhold  payment  for  incomplete  or  unaccept- 
able work. 

b.  Procedure 

•  If  agency  evaluation  of  the  contractor's  work  indicates  that  the  approach  being 
used  will  not  produce  the  needed  software,  and  If  the  contractor's  work  cannot 
be  redirected,  the  contract  should  be  terminated  to  minimize  losses. 

c.  Possible  Ways  of  Accomplishing  the  Procedures 

•  Satisfactory  performance  criteria,  and  acceptance  testing  criteria  should  be 
used  to  identify  work  that  does  not  meet  contract  requirements. 
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•  Payment  to  the  contractor  should  be  reduced  by  the  amount  of  any  deliver- 
ables (e.g.,  documentation)  specified  in  the  contract  but  not  produced. 

C       PHASE:  PROPOSAL  EVALUATION 

I.  OBJECTIVE 

•  Ensure  that  a  sl<illed  and  responsible  contractor  is  selected. 

a.  Method 

•  Only  contractors  with  verified  technical  skill  and  financial  responsibility 
should  be  selected  to  develop  custom  software  contracts. 

b.  Procedure 

•  Agencies  should  evaluate  each  proposal  in  terms  of  relevant  factors,  including 
those  listed  below,  and  including  considerations  of  feasibility,  comparison  to 
other  proposals'  prices  and  schedules  and,  if  possible,  comments  solicited  from 
the  contractor's  prior  customers. 

c.  Possible  Ways  of  Accomplishing  the  Procedures 

•  Include  in  the  request  for  proposal  a  provision  for  agency  representatives  to 
visit  the  contractor  site  to  investigate  and  evaluate  various  factors,  including 
financial  position,  technical  capability,  and  labor  resources. 

•  Try  to  determine  whether  or  not  the  contractor's  capabilities,  qualifications, 
and  experience  are  commensurate  with  the  complexity  of  the  problem. 

•  Try  to  determine  whether  or  not  the  contractor's  staff  has  experience  with  the 
software  and  hardware  to  be  used  during  development. 
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•  Consider  the  effect  on  the  contract  schedule  of  the  contractor's  experience 
(e.g.,  delay  in  training  the  contractor's  staff). 

D,       PHASE;  PERFORMANCE  PERIOD 

I.  OBJECTIVE 

•  The  agency  should  manage  the  contract  during  execution  in  a  manner  that 
contributes  to  its  success. 

a.  Methods 

•  The  agency  should  provide  all  of  its  deliverables  to  the  contractor  within  the 
specified  timeframes. 

•  Management  should  create  an  environment  within  the  agency  that  supports  the 
contractor's  efforts. 

b.  Procedures 

•  Such  things  as  studies,  requirements,  and  general  designs  to  be  provided  by  the 
agency  should  be  provided  promptly  so  that  the  contractor  is  not  delayed. 

•  if  provided  by  the  agency,  such  work  products  should  be  complete,  accurate 
and  provide  a  basis  for  the  contractor's  work. 

c.  Possible  V/ays  of  Accomplishing  the  Procedures 

•  Management  should  be  involved  at  high  enough  levels  to  insure  cooperation  by 
all  agency  personnel.  If  there  is  disagreement  about  the  new  development 
effort  among  various  factions  within  the  agency,  it  should  be  resolved  in-house 
by  management  and  not  left  for  the  contractor  to  encounter. 
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•  If  possible,  the  same  agency  staff  should  be  kept  on  the  project  throughout  the 
contract. 

•  A  single  focal  point  should  be  established  within  the  agency  to  deal  with  the 
contractor. 

•  An  open  line  of  comnnunication  should  be  maintained  with  the  contractor. 
However,  undocumented  informal  communication  can  lead  to  cost  overruns. 
Any  changes  in  the  scope  of  work  must  be  handled  by  formal  modification  of 
the  contract  document. 

2.  OBJECTIVE 

•  Monitoring  the  contractor's  progress  to  ensure  that  critical  milestones  are  met 
and  to  approve  work  segments. 

a.  Methods 

•  Use  timeframe  criteria  that  are  established  to  determine  whether  the  contrac- 
tor's development  is  on  schedule. 

•  All  work  should  be  reviewed  at  the  end  of  each  completed  phase  to  determine 
if  it  conforms  with  contract  specifications. 

b.  Procedures 

•  Check  progress  against  dates  on  the  milestone. 

•  Be  alert  for  signs  of  problems  in  the  contractor's  progress  reports. 

•  Conduct  any  independent  reviews  of  the  contractor's  operation  that  are 
considered  necessary. 
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c.        Possible  Ways  of  Accomplishing  the  Procedures 

•  Consider  how  well  agency  requirements  are  met. 

•  Decide  if  the  contractor's  approach  is  technically  feasible. 

•  Obtain  users'  evaluation  of  contractor's  work. 

•  Render  timely  management  decisions  on  all  alternatives  presented  by  the 
contractor. 

•  Once  a  phase  is  approved,  freeze  that  phase  until  development  Is  complete  to 
stabilize  the  base  for  succeeding  phases. 

•  Acceptance  testing  may  be  applied  to  completed  phases  as  well  as  at  the  end 
of  the  development  effort. 

3.  OBJECTIVE 

•  Do  adequate  testing  to  ensure  that  the  product  meets  contract  specifications, 
a.        Method  I 

•  Make  sure  acceptance  test  criteria  are  still  current. 

•  Maintain  adequate  control  over  the  testing  operation. 

1.  Procedure 

•  if  test  criteria  and  data  were  developed  in  the  beginning,  make  certain  they 
have  been  revised  to  incorporate  system  changes,  if  any. 
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ii.        Possible  Ways  of  Accomplishing  the  Procedures 


•  Observe  or  participate  in  software  test. 

•  Adequately  analyze  the  test  results. 

•  Docunnent  all  errors  revealed  by  the  test. 

•  Require  the  contractor  to  correct  all  discrepancies  as  a  condition  for  final 
payment. 

•  Follow  up  on  all  discrepancies  to  make  sure  they  are  corrected  before  the 
system  is  accepted. 

b.        Method  II 

•  Ensure   that   personnel   responsible   for   testing   the  system   have  adequate 
technical  expertise. 

i.  Procedures 

•  Assign  qualified  agency  staff  with  systems,  data  processing,  and  performance 
evaluation  expertise  to  test  software. 

•  If  the  agency  does  not  have  the  expertise  to  adequately  evaluate  the  software, 
arrange  for  an  independent  evaluation  by  outside  sources. 

4.  OBJECTIVE 

•  To  exercise  ail  remedies  on  behalf  of  the  agency  in  case  the  contractor  fails  to 
perform. 
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a.  Method 


•  Full  payment  should  not  be  made  to  the  contractor  until  it  is  determined  that 
the  software  meets  contract  specifications. 

b.  Procedure 

•  Exercise  the  contract  recourse  provisions  discussed  under  contract  controls  to 
the  degree  that  nonperformance  was  encountered. 

E.       PHASE:  POST-CONTRACT 

I.        OBJECTIVE  ■ 

•  Identify  both  good  and  bad  aspects  of  the  software  procurement. 

a.  Method 

•  Make  a  followup  analysis  of  the  software  development  procurement  contract 
to  evaluate  contracting  practices,  record  lessons  learned,  and  evaluate  user 
satisfaction  with  the  product. 

b.  Procedures 

•  Identify  practices  that  are  weak  and  need  to  be  changed. 

•  Identify  and  retain  practices  that  produced  good  results. 

•  Identify  additional  agency  guidelines  that  need  to  be  developed  and  imple- 
mented. 

•  Evaluate  user  satisfaction  with  the  software. 
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•  Record  the  actual  amount  of  maintenance  programming  work  that  is  needed 
soon  after  the  software  is  put  into  production. 

•  Retain  performance  data  on  the  individual  contractor  for  future  reference. 
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APPENDIX  G:         INTERVIEW  PROGRAM 


On-Site  Telephone 

Company  Type  Interviews  interviews  Total 

Amnnunition  Plants  7-7 

EPCS  Vendors  6                  9  !5 

EPCS  Users  5                  9  14 

Maintenance  Service  Companies  2                   1  3 

Other  Vendors/Users  -                   5  5 

TOTAL  20                24  44 
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CATALOG  NO.    |V|lVl|DiCi  | 


INTRODUCTION 
AMMUNITION  PLANT  SITE  VISIT 


INPUT  has  been  retained  by  the  U.S.  Army  to  develop  a  maintenance  format,  and 
program  for  electronic  process  controiled  ammunition  plants.  This  program  is  to  be 
based  upon  the  information  available  on  both  current  and  future  commercial  systems 
combined  with  current  systems  information  in  various  ammunition  plants  around  the 
United  States. 

This  outline  has  been  prepared  to  ensure  that  similar  information  is  gathered  at  each 
plant  site  and  to  ensure  that  no  important  items  are  omitted.  Items  may  be  added 
that  are  felt  to  be  pertinent  to  the  subject.  Reasonably  detailed  systems  information 
is  requested  so  that  the  subsequently  proposed  maintenance  program  will  be  most 
effective  and  appropriate  for  this  site. 
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CATALOG  NO.   lYIMiDICI  I T1 


AMMUNITION  PLANT  SITE  DISCUSSION  OUTLINE 

PART  I 


I.        Plant  description. 

A.  Plant  name  and  address. 

B.  Type  of  plant. 

C.  How  plant  integrates  with  other  plants. 

D.  History  of  plant. 

1.  State  of  construction/operation. 

2,  Vendors  involved. 


3*        Level  of  autonnatlon. 


4".        Start-up  problems. 


5.        Dates  of  construction,  equipment,  installation,  etc. 
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CATALOG  NO.    tVlMlnlrl   i  I 


AMMUNITION  PLANT  DISCUSSION  OUTLINE 

PART  2 


II.       EPC  system  description. 


A.       Overview  description. 


B.       Block  diagram. 


I.  Plant. 


2.  EPCS. 
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3.  Software. 


CATALOG  xNO.  IyImIDIC 


C.  Hardware. 


I.        Specific  computer  models. 


2.        Programmable  controllers. 


3.        Interface  models. 


4.        I/O  equipment. 
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5.        Communications  equipment. 


CATALOG  NO.  lYlMlD 


6.        Instruments,  sensors,  flowmeters,  measuring  devices. 


7.        Other  equipment. 


8.        Documentation  available. 


D.  Software. 


i.        Standard  programs. 
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CATALOG  :;o. 


VIVJDIC 


a.        Computer  vendor  supplied. 


b.        EPCS  vendor  supplied. 


c.  Documentation. 


d.        Storage  media. 


Custom  softv/are. 


a.  Vendor. 
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b.  Documentation. 


c.  Standards. 


E.  Environnnent. 


I.  Operating. 


a.  Temperature. 


b.  Humidity. 


CATALOG  NO.  fYlMIDIC 
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c.  Atmosphere. 


CATALOG  NO. 


d.        Other  factors— (example:  vibration). 


Dormant. 


a.  Temperature. 


b.  Humidity. 


c.  Atmosphere. 
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d.        Other  factors— (condition  of  neighboring  facilities). 


III.      Operating  experience. 


A.       Production  running. 


I.        Time  to  reach  maximum  production  rate. 


a.        Maximum  potential. 


b.        Maximum  achieved. 
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CATALOG  NO. 


lYiMiprj  I  I 


c.        Average  rate. 


2.        Downtime  experience. 


3,        Reasons  for  failure. 


a.        Tracking  failures. 


b.  Analysis. 


4.        MTTR— (possibilities  for  reduction). 
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CATALOG  NO.    iYIMIDICI   I T1 
5.        MTBF— (possibilities  for  increase). 


6.        Plans  for  improvement. 


B.       Maintenance  procedures. 


i.        Overall  plan. 


2,        Actual  implementation. 


3.        Changes  based  upon  system  performance. 
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Current  posture. 


CATALOG  NO.  IviMiU 


5.        Future  plans. 


6,        Reliability  estimating  procedures. 


7.        Failure  prediction  techniques. 


8.        Critical  areas. 


IV.       Reactivation  experience. 
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Reactivation  history. 


Reactivation  plans. 


Testing. 


2.  Mobilization. 


Dormant  storage  conditions. 


CATALOG  NO.    lYlMlDlri  fl 


I.  Computers. 
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Controllers. 


CATALOG  NO. 


Measuring  and  communicating  devices. 


a.  Electromechanical. 


b.  Optical. 


c.  Electronic. 


d.  Other. 
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4.        Magnetic  media. 


CATALOG  NO.  [YMD" 


5.  Software. 


6.        Documentation  storage. 


V.       Parts  inventory  management. 


A.       Parts  listings—detail. 


i.        Detailed  vendor  listing. 
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CATALOG  NO.    |Y|M|D|C|  I 


B.        Spare  parts  program. 


C.       Vendor  support. 


i.        Experience  with  vendors. 


2.        Active—time  period. 


3.        Passive— time  period. 


D.       Alternate  parts  sources. 
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E.       Compatible  parts  assessment  program. 


CATALOG  NO. 


inmifafiiMa 


VI.       Personnel  skills  program. 


A.       Current  skills  assessment. 


Operating  personnel. 


2.        Maintenance  personnel. 


a.  Hardv/are. 
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b.  Software. 


CATALOG  NO. 


Em 


B.       Reactivation  skills  required. 


!.  Hardware. 


2.  Software. 


C.       Personnel  tracking  program. 


D.       Personnel  training  program. 
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Frequency. 


CATALOG  NO. 


m\c\  1  [ 


Techniques  used  or  planned. 


Trained  cadre  required. 


Training  vendors. 


Sources  of  new  personnel. 


Personnel  problems  anticipated. 
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CATALOG  NO.    lYlMlDlrl  iT 

a.        Planned  solutions. 


VII.      Operating  contractor's  recommendations. 


A.       Improvement  of  operating  environment. 


B.       Relationship  with  system  vendors  and  subvendors. 


C.       Maintainability  requirements. 


D.       Reactivation  procedures. 
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CATALOG  NO.    IyiMIDICI   I  I 


E. 


Equipment  storage. 


F.       Spare  parts  management. 


G.  Personnel. 


H.       Training  program. 


I.  Other. 


* 

THANK  YOU  FOR  YOUR  COOPERATION 
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CATALOG  NO.    lY  I  Ml  HI  P. 


ELECTRONIC  PROCESS  CONTROL  SYSTEM  USER  DISCUSSION  GUIDE 

INPUT  has  been  retained  by  the  U.S.  Army  Armannent  Research  and  Development 
Command  to  develop  information  to  establish  a  methodology  for  the  repair  and 
maintenance  of  electronic  process  control  systems  that  will  be  inactive  over  long 
periods  of  time. 

The  emphasis  of  this  study  is  on  the  electronic  control  portion  of  the  systems— the 
computer,  the  lines,  the  controller,  the  I/O  subsystem,  the  sensors  and  the  software. 

The  U.S.  Army  is  interested  in  user's  experience  with  respect  to  reliability  and 
maintainability  of  commercial  electronic  process  control  systems.  Special  interest  is 
directed  tov/ards  those  systems  that  are  shut  down  for  varying  periods  of  time  due  to 
the  closing  of  a  production  plant,  the  decommissioning  of  a  battleship,  or  the 
cocooning  of  an  aircraft.  Any  suggestions  concerning  the  storage,  reactivation,  or 
maintainability  of  such  systems  are  welcome.  Comments  in  the  areas  of  spare  parts 
management  and  personnel  development  are  also  welcome. 

In  return  for  your  assistance  with  this  research,  a  summary  of  the  key  findings  will  be 
sent  to  you  at  the  conclusion  of  the  study. 
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CATALOG  NO.    |Y  |M|  DICI  IT 
ELECTRONIC  PROCESS  CONTROL  SYSTEM  USER  DISCUSSION  GUIDE 

General  information. 


A.       System  background. 


I.        Year  of  design. 


2.        Date  of  installation  (acceptance). 


3.        System  supplier/designer. 


4.        Software  supplier. 
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5.        System  diagram. 


CATALOG  NO.    |V  |M|D|C|   }  "J 


B.       System  hardware. 


Computer  manufacturer  and  model. 


2.        Memory  size. 


3.        Communication  lines. 


4.        I/O  equipment. 
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CATALOG  NO.   lYIMIDICI  H 


5.        Mass  storage— type,  size,  speed. 


6.        Sensors— number  and  type. 


7.        Programmable  controllers. 


C.  Software. 


!.        Standard  software  source. 


2.        Custom  software  source. 
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3.        Block  diagram. 


D.       Back-up  system. 


I.  Warm/cold. 


2.        Critical  areas. 


3.  Equipment. 


CATALOG  NO.   IyImIdI  C.\   I  7 


4.        Data  storage. 
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CATALOG  NO. 


crrn 


5.  Communications. 


6.        Other  considerations. 


E.       Fail-safe  considerations. 


Design. 


2.  Procedures. 


3.  Other. 
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F.  Support. 


CATALOG  NO.   lY  IMID 


C 


n 


1.        From  systems  designer/contractor. 


2.        From  computer  manufacturer. 


3.        From  software  vendor. 


il.       Reliability  experience. 


A.  History. 
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Initial  start-up  problems. 


2.        Problems  after  production  running. 


3.        Failure  prediction  algorithms  used. 


Method  for  handling  problems. 


I.        Failure  reporting  and  tracking  system. 


2.        Feedback  loop  for  improving  system. 
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MTBF. 


CATALOG  NO. 


irm 


4.  MTSF. 

5.  MTTR. 

6.  Operator's  responsibilities. 

7.  System  designer's  responsibilities. 

8.  Equipment  manufacturer's  responsibilities. 
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CATALOG  NO.    lYIMI  DICI   I  Tl 


9.        Software  designer's  responsibilities. 


111.  Maintainability. 


A.       Design  features. 


B.       Preventive  maintenance  procedures. 


Planned. 


2.        Reasons  for  modification. 
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3.        Current  procedures. 


CATALOG  NO.    lYlMlnlfl   I  T 


Personnel  for  maintenance. 


I.        Skills  required. 


2.        Source  of  personnel. 


3.        Amount  of  core  cadre  necessary. 


4.        Training  provided. 
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5.        Future  problems/solutions. 


CATALOG  NO. 


Spare  parts  management. 


A.  Inventory. 


I.        How  determined. 


2.        Frequency  of  change. 


3.        Current  and  average  dollar  value  of  inventory. 
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Storage  conditions/packaging. 


5.        Average  storage  time  for  spares. 


6.        Maximum  storage  time  for  any  part. 


7.        Recycling  testing  procedures. 


8.        Usage  experience  as  a  function  of  storage  time. 


9.        Handling  of  "out  of  stock"  conditions. 
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CATALOG  NO.    lYIMIDICi   \  1 


B.  Obsolescence. 


1.  Experiences. 


2.        Suppliers'  policies. 


3.        Alternate  sources. 


4.        Compatible  parts  assessment  program. 


5.  Suggestions. 
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CATALOG  NO.   lYiMIDICI  iT 
V.       Shutdown,  dormancy  and  reactivation  experience. 


A.  History. 


I.        Shutdown  dates  and  reasons. 


2.        Length  of  dormant  periods. 


3.        Dates  of  reactivation. 


B.       Conditions  of  dormancy. 
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CATALOG  NO.    |Y  |M|  U|L  j  fl 


Packaging. 


2.  Environmental, 


3.  Other. 


C.  Reactivation. 


Plan. 


2.  Procedures. 
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Precautions  taken. 


CATALOG  NO. 


Y 


Problems  encountered. 


Solutions. 


Feedback  loop. 


Other  comments. 
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CATALOG  NO.    lYlMiniCI  l"T 


VI. 


Future. 


A.       Technological  changes  that  affect  process  control  systems  reliability 
and  maintainability. 


B.       If  you  did  it  over  again,  what  would  be  done  differently? 


C.       Advice  for  EPC  system  builders  and  users. 


THANK  YOU  FOR  YOUR  COOPERATION 
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ELECTRONIC  PROCESS  CONTROL  SYSTEM  VENDOR  DISCUSSION  GUIDE 


INPUT  has  been  retained  by  the  U.S.  Army  Armament  Research  and  Development 
Command  to  develop  information  to  establish  a  methodology  for  the  repair  and 
maintenance  of  electronic  process  control  systems  that  will  be  inactive  over  long 
periods  of  time. 

The  emphasis  of  this  study  is  on  the  electronic  control  portion  of  the  system—the 
computer,  the  lines,  the  I/O  subsystem,  the  sensors  and  the  software. 

The  U.S.  Army  is  interested  In  which  developments  in  future  systems  will  tend  to 
increase  systems  reliability  and  contribute  significantly  to  the  ease  of  maintaining 
these  systems  during  long  periods  of  inactivity.  Any  suggestions  concerning  present 
or  future  maintenance  methodologies,  spare  parts  management,  and  personnel 
development  ore  welcome. 

In  return  for  your  assistance  with  this  research,  a  summary  of  the  key  findings  will  be 
sent  to  you  at  the  conclusion  of  the  study. 
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ELECTRONIC  PROCESS  CONTROL  SYSTEM  VENDOR  DISCUSSION  GUIDE 
General. 


A,       Size  of  electronic  process  control  industry. 


I.        Number  of  major  vendors. 


2.        Total  industry  dollar  volume  of  business— 1 979. 


3.        Growth  over  last  10  years. 


B.        Major  vendors. 
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I.        Name  and  rank  (sales  in  EPCS  business  only). 


C.       Major  user  industries. 


D.       Third  party  nnaintenance  companies. 


II.       Currently  available  systems. 


A.  Reliability. 


I.        Considerations  in  design. 
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a.        How  are  subunits  selected. 


CATALOG  NO. 
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2.        Trade  offs  available. 


3.  MTBF. 


a.  Standard. 


b.        Means  of  maximizing. 


c.        Cost  of  maximizing. 


d.        Maximum  achievable. 
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e.        Failure  prediction  algorithms. 


f.        Comments— (example:  Major  influencing  factors.) 


MIL-SPEC  components. 


a.        Cost  versus  reliability. 


b.        Availability  for  EPC  systems. 


Other  packaging  techniques  for  components. 
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6.        System/subsystem/device  weak  points. 


a.  Reason. 


b.        Design  to  support  them. 


c.        Other  considerations  or  comments. 


7.        Off  the  shelf  components  versus  custom  components. 


Maintenance  procedures. 
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Who  maintains. 


a.        Customer's  responsibilities. 


b.        System  supplier's  responsibilities. 


c.        Subsystem  supplier's  responsibilities. 


d.        Software  maintenance  and  trouble  shooting  responsibilities. 


2.        Preventive  maintenance  routine. 
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a.        Differences  by  type  of  system. 


3.        On-site  maintenance. 


a.        Personnel  skills  required. 


b.        First  line  of  action. 


c.        Second  line. 


d.        Third  party  maintenance  company  usage. 
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4.        Off -site  maintenance. 
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a.  Prevaiency. 


b.        Reasons  for. 


c.  Trends. 


5.        Maintenance  cost  considerations. 


a.        T  &  M  versus  contract. 
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b.        On-site  versus  depot. 


c.        User  self-maintenance  trends. 


C.       Documentation  (hardware  and  software). 


Documentation  normally  supplied. 


a.  Levels. 


b.  Maintenance. 
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c.  Media. 
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2.        Documentation  reasonable  to  expect  (by  contract  and  at  what 
cost.) 


D.       Spare  parts  inventory  nnanagennent. 


I.        Standard  procedures  for  determining  necessary  inventory. 


2.        Factors  that  tend  to  modify  standard  practices. 


3.  Packaging. 


-  385  - 


INPUT 


CATALOG  NO.  |Y|M|D 

4.  Obsolescence. 


5.        Alternate  sources  of  parts. 


6.        Identification  and  assessment  of  compatible  parts. 


7.        Spore  parts  time  guarantee. 


E.        Customer  support. 


I.        Prior  to  system  acceptance. 
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2.        Post-system  acceptance. 


Q.        Type  of  support. 


b.        Level  of  support. 


c.        Active  support— how  long. 


d.        Passive  support— inow  long,  type. 


e.        How  long  is  reasonable  by  contract. 
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F.        The  dormant  state. 


I.        System  preparation  procedures. 


2.        Suitable  packaging. 


a.  Computer. 


b.        1/0  equipment. 


c.        Lines,  cables. 
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d.        Sensors,  flowmeters,  other  measuring  devices. 


e.        Magnetic  disks,  tape,  etc. 


f.        Paper  tape. 


g.  Documentation. 


h.        Spare  parts. 


i.        Other  system  parts. 
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3.        Environmental  considerations. 


a.  Temperature. 


b.  Humidity. 


c.  Atmosphere. 


d.  Other. 


4.        Additional  considerations  or  suggestions. 
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Reactivation. 


i.        Unpacking  procedures. 


3.        Activation  precautions. 


a.        Sensitive  system  areas. 


4.        Failure  tracking. 


5.        Failure  reporting. 
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6.        Suggested  procedures  to  prevent  repetition. 


7.        Suggested  reactivation  frequency. 


8.        Procedural  differences  as  a  function  of  length  of  dormant  period. 


H.  Personnel. 


Skills  necessary  for  system  m.aintenance. 


2.        Skills  necessary  for  reactivation. 


TOO 
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3.        Sources  of  suitable  personnel. 


4.        Trained  cadre  required  for  reactivation. 


5.        Suggested  training  methods  and  procedures. 


a.        Availability  of  training  materials  at  novice  level. 


6.        Personnel  tracking  systems. 


II.       Future  systems. 
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a.        Trends  in  process  control  industry. 


Reliability  related. 


2.        Maintainability  related. 


3.        Personnel/training  related. 


B.        Areas  of  technological  change  that  will  affect; 


I .        Systems  design. 
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Systems  reliability. 
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3.        Systems  maintainability. 


4.        Component  reliability. 


5.        Long-term  storage  of  subsystems  and  components. 


f 

6.        Packaging  of  spares. 


7.        Parts  interchangeability  and  compatibility. 
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8.        Diagnostic  procedures,  techniques  and  capability. 


9         Software  production  and  maintenance. 


10.      Documentation  production  and  storage. 
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C.       Suggested  maintenance  framework  for  dormant  EPD  systems. 


D.       Additional  comments  and  advice. 


THANK  YOU  FOR  YOUR  COOPERATION 
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